The elements of enlightenment

The elements of enlightenment

[Ray Andrews]

I  think I've finally actually figured out the reason that lines aren't 
elements and elements aren't lines even though they can print exactly 
the same:

     $ list=( $( setopt ) )

... We get word splitting and the option and the value 'print -l' on 
different lines and '$#list' is the count of the number of words/lines 
namely twice the number of options.  Good, fine, and understood.  But I 
want the option and it's value on the same line, just as if 'setopt' was 
executed at CLI.  By double quoting: ( "$( setopt )" ) we seem to have 
solved the problem, it prints correctly.  But trouble lurks in the 
shadows.   I do this:

     $  list=( "${list[@]/ off/${red} off${nrm}}" )

... And I'm baffled that only the very first 'off' is colorized. Why?  
Because '$#list' = 1!  It looks like we have each line as a separate 
element but we don't.  It only looks that way because the newlines in 
the output are still in there, they aren't 'print -l' newlines, they're 
newlines in the data itself!! They look the same but they are *not* the 
same.  To really get what it looks like I have I must first:

     $ list=( ${(f)list} )

NOW our element count is what we want it to be -- one per line and every 
'off' get colored.  Newlines aren't elements and elements aren't 
newlines.  Finally!  I'll never be fooled by appearances again.  Do I 
have this right?  Only took ten years :(

Re: The elements of enlightenment

[Roman Perepelitsa]

On Mon, Dec 5, 2022 at 4:58 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> I  think I've finally actually figured out the reason that lines aren't
> elements and elements aren't lines even though they can print exactly
> the same:
>
>      $ list=( $( setopt ) )

This is the standard way of splitting the output of a command into
lines and dropping empty lines:

  list=( ${(f)"$(setopt)"} )

This requires KSH_OPTION_PRINT, just like your code. It may not be the
best idea to write code that deals with options in a way that requires
specific options.

It's unlikely that you need to capture the output of setopt in the
first place given that there is $options associative array.

  () {
    local k v
    local -A color=(on 32 off 33)
    for k v in ${(kv)options}; do
      printf '%-20s = \e[%sm%s\e[0m\n' $k ${color[$v]} $v
    done
  }

Dealing with options as an associative array is a lot easier than
dealing with text.

Roman.

Re: The elements of enlightenment

[Bart Schaefer]

On Mon, Dec 5, 2022 at 8:24 AM Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> This requires KSH_OPTION_PRINT, just like your code. It may not be the
> best idea to write code that deals with options in a way that requires
> specific options.

The potential effects of LOCAL_OPTIONS already make this complicated
enough.  Cf. hoops the completion system jumps through.

Re: The elements of enlightenment

[Ray Andrews]

On 2022-12-05 08:24, Roman Perepelitsa wrote:
> This is the standard way of splitting the output of a command into
> lines and dropping empty lines:
>
>    list=( ${(f)"$(setopt)"} )
>
The thing is I finally really understand it.  I thought newlines always 
broke elements.  I thought that what you saw was what you had.
>    () {
>      local k v
>      local -A color=(on 32 off 33)
>      for k v in ${(kv)options}; do
>        printf '%-20s = \e[%sm%s\e[0m\n' $k ${color[$v]} $v
>      done
>    }

That's most educational, I had no idea you could do that 'for k v in' 
double capture.  And that's a cool demonstration of the use of an 
associative array -- the name can be called by a variable. Powerful.  
But, since setopt is a builtin anyway, why not just use it as is?  I 
love replacing binaries with native zsh code, but setopt is part of the 
family anyway so why not just take it's output?  The function I'm 
playing with requires the array as I have it, just getting the printout 
isn't enough.  I tweaked your code like this:

local k v
local list=()
local -A color=(on ${red} off ${nrm})        # My color variables do the 
predictable thing.
     for k v in ${(kv)options}; do
         list+=$( printf '%-20s %s%s\e[0m\n' $k ${color[$v]} $v )
     done

... to get my list.  The equivalent as I now have it is:

local list=( "$( setopt )" )
list=( ${(f)list} )
list=( "${list[@]/ off/${red} off${nrm}}" )    # red = 'off' here cuz 
that's the way Sebastian had it.

Which is best?  Probably hardly matters, but still, it's good to know.  
printf is heavy, no?  setopt might be lighter?

Re: The elements of enlightenment

[Bart Schaefer]

On Mon, Dec 5, 2022 at 11:35 AM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> Which is best?  Probably hardly matters, but still, it's good to know.
> printf is heavy, no?  setopt might be lighter?

Any time you use $(...) [with one exception] zsh is going to fork an
additional process.  So the circumstances in which that would be
"lighter" than doing anything entirely inside the current shell are
vanishingly small.

The exception is that $(<file) reads the file without forking.

Re: The elements of enlightenment

[Roman Perepelitsa]

On Mon, Dec 5, 2022 at 8:36 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> But, since setopt is a builtin anyway, why not just use it as is?

Parsing things from text is usually more complex than dealing with the
things that are already in a decent data structure. Your specific task
may be an exception if you really want to produce the same output as
`setopt` but with "off" in red. In this case it may be even desirable
to manipulate the output of `setopt` rather than reconstructing it
manually.

> printf is heavy, no?  setopt might be lighter?

Neither printf nor setopt are heavy. Forks are extremely inefficient
though. The cost of a fork is almost astronomical compared to simple
operations. The following two lines achieve the same thing:

    var=$(printf hello)
    printf -v var hello

On my machine the first line is 1500 times slower than the second. You
need to execute this line only 100 times in order to have the total
run time of a script in excess of 1 second. Imagine if every line of a
script was this slow.

Roman.

Re: The elements of enlightenment

[Ray Andrews]

On 2022-12-05 11:58, Roman Perepelitsa wrote:
>
> Neither printf nor setopt are heavy. Forks are extremely inefficient
> though.
... and '$( ... )' is a fork?  Yes, clearly by your example.  Yeah, it 
spawns another shell, yes?
>      var=$(printf hello)
>      printf -v var hello
>
> On my machine the first line is 1500 times slower than the second.

Holy cow, 1500 X ??

I like efficiency in principle even if in practice it's miliseconds. But 
I have to keep appending to the list for each line, can the 'print -v 
var' method handle that?  Not important, but it would be satisfying to 
do the whole thing in the current shell.

Re: The elements of enlightenment

[Roman Perepelitsa]

On Mon, Dec 5, 2022 at 10:21 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> On 2022-12-05 11:58, Roman Perepelitsa wrote:
> >
> > Neither printf nor setopt are heavy. Forks are extremely inefficient
> > though.
>
> ... and '$( ... )' is a fork? [...] it spawns another shell, yes?

It invokes fork(2). See `man fork`.

> I like efficiency in principle even if in practice it's miliseconds.

That's why I tried to quantify it in related terms: on my machine
executing that thing 100 times takes over a second.

    % time ( repeat 100 var=$(printf hello) )
    user=0.00s system=1.06s cpu=90% total=1.178

> I have to keep appending to the list for each line, can the 'print -v
> var' method handle that?

See `run-help printf` to see what `printf -v` does.

Roman.

Re: The elements of enlightenment

[Ray Andrews]

On 2022-12-05 13:49, Roman Perepelitsa wrote:
> That's why I tried to quantify it in related terms: on my machine
> executing that thing 100 times takes over a second.
>
>      % time ( repeat 100 var=$(printf hello) )
>      user=0.00s system=1.06s cpu=90% total=1.178

>      for k v in ${(kv)options}; do
> 	# 100X slower!!!
> 	#list+=$( printf '%-20s %s%s\e[0m\n' $k ${color[$v]} $v )
> 	printf -v line '%-20s %s%s\e[0m' $k ${color[$v]} $v
> 	list+="$line"
>      done
>
Even buried back in the entire function, where the difference is 
somewhat swamped by the time the other code takes to run,  it's still 
100X faster doing it your way.  Sheesh, it's quite astonishing. I 
probably wouldn't understand the answer anyway but I hafta wonder why 
this forking is so necessary.  But for now 'print -v var' is my new friend.

Re: The elements of enlightenment

[Bart Schaefer]

On Mon, Dec 5, 2022 at 3:52 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> probably wouldn't understand the answer anyway but I hafta wonder why
> this forking is so necessary.

I wrote to you back on October 7:

> $(...) spawns a subshell because stdin/stdout text is passed from one
> process to another using pipes, and if a process has a pipe to itself
> it can deadlock trying to read something it hasn't written yet (or
> write something there isn't room for because not enough has been read
> yet).

Re: The elements of enlightenment

[Ray Andrews]

On 2022-12-05 16:17, Bart Schaefer wrote:
>
>> $(...) spawns a subshell because stdin/stdout text is passed from one
>> process to another using pipes, and if a process has a pipe to itself
>> it can deadlock trying to read something it hasn't written yet (or
>> write something there isn't room for because not enough has been read
>> yet).

Yeah I remember we discussed it.  It doesn't stick because I don't 
really understand it.  As just now, one can do it the forking way or one 
can do it in the one shell, and the later works fine, so the former is 
obviously not needed.  But, not knowing the guts, I can only make the 
most superficial comments.  As you say, traffic control could mandate 
things that might not be desirable from the point of view of speed.

Re: The elements of enlightenment

[Roman Perepelitsa]

On Tue, Dec 6, 2022 at 12:53 AM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> I hafta wonder why this forking is so necessary.

It's necessary because it's user-visible. Some code relies on the PID
within $(...) to be different from the parent. This is mostly relevant
for signals. In addition, changes to parameters and other shell state
within $(...) have no effect on the parent shell, which doesn't
necessarily require forking but is made easier by it.

That said, it's totally possible to extend zsh with something like $[
... ] which would work like $( ... ) but without forking. It would run
the commands in another thread within the same process. The main
thread would be blocked on reading from the pipe until all commands
are executed.

Roman.

Re: The elements of enlightenment

[Bart Schaefer]

On Tue, Dec 6, 2022 at 10:56 AM Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> That said, it's totally possible to extend zsh with something like $[
> ... ] which would work like $( ... ) but without forking. It would run
> the commands in another thread within the same process.

That requires enabling support from threading libraries like pthreads,
which bring along their own set of availability and synchronization
challenges.

Re: The elements of enlightenment

[Roman Perepelitsa]

On Tue, Dec 6, 2022 at 8:45 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> On Tue, Dec 6, 2022 at 10:56 AM Roman Perepelitsa
> <roman.perepelitsa@gmail.com> wrote:
> >
> > That said, it's totally possible to extend zsh with something like $[
> > ... ] which would work like $( ... ) but without forking. It would run
> > the commands in another thread within the same process.
>
> That requires enabling support from threading libraries like pthreads,
> which bring along their own set of availability and synchronization
> challenges.

Yeah, it's probably non-trivial. Thankfully, it wouldn't require
inserting mutex locks in random places because the parent shell will
never be executing concurrently with the child. It'll be blocked while
the child is running. Signal handling might need changes though. I
don't know this part of the code well enough to know whether they
would be big or not.

I'm not suggesting that it's a good idea to implement $[ ... ]. I'm
just musing on the topic of capturing stdout from zsh code without
forking.

Roman.

Re: The elements of enlightenment

[Bart Schaefer]

On Tue, Dec 6, 2022 at 12:01 PM Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> Yeah, it's probably non-trivial. Thankfully, it wouldn't require
> inserting mutex locks in random places because the parent shell will
> never be executing concurrently with the child.

I generally hate slippery-slope arguments, but ... as soon as we have
$[...], someone is going to want "coproc" and <(...) threads too.

Re: The elements of enlightenment

[Roman Perepelitsa]

On Tue, Dec 6, 2022 at 9:07 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> On Tue, Dec 6, 2022 at 12:01 PM Roman Perepelitsa
> <roman.perepelitsa@gmail.com> wrote:
> >
> > Yeah, it's probably non-trivial. Thankfully, it wouldn't require
> > inserting mutex locks in random places because the parent shell will
> > never be executing concurrently with the child.
>
> I generally hate slippery-slope arguments, but ... as soon as we have
> $[...], someone is going to want "coproc" and <(...) threads too.

And `&` for threads obviously. Yeah, any of those are very difficult.

In practice I don't think $[ .. ] would be super useful. One can use
`printf -v` instead of $(printf) and the same applies to pretty much
all instances of invoking a builtin and capturing its stdout: there is
always a way to get the same data without printing it. I know of only
one exception. To the best of my knowledge, the only way to get
keybindings is to invoke bindkey and capture stdout.

Roman.

Re: The elements of enlightenment

[Lawrence Velázquez]

On Tue, Dec 6, 2022, at 1:55 PM, Roman Perepelitsa wrote:
> That said, it's totally possible to extend zsh with something like $[
> ... ] which would work like $( ... ) but without forking. It would run
> the commands in another thread within the same process.

FWIW, ksh has a ${ list;} construct that avoids a subshell:

	$ cd / && pwd
	/
	$ : "${ cd /tmp;}" && pwd
	/tmp
	$ : "${ cd /usr }" && pwd
	/usr

Amusingly, the archaic $[...] is still used for arithmetic expansion.
It's even documented!  (Other shells have memory-holed it.)

-- 
vq

Re: The elements of enlightenment

[Lawrence Velázquez]

On Tue, Dec 6, 2022, at 6:52 PM, Lawrence Velázquez wrote:
> FWIW, ksh has a ${ list;} construct that avoids a subshell:
>
> 	$ cd / && pwd
> 	/
> 	$ : "${ cd /tmp;}" && pwd
> 	/tmp
> 	$ : "${ cd /usr }" && pwd
> 	/usr
>
> Amusingly, the archaic $[...] is still used for arithmetic expansion.
> It's even documented!  (Other shells have memory-holed it.)

To clarify my sloppy editing:  The last paragraph is about zsh.
Other shells also support legacy $[...] but pretend it doesn't
exist.

-- 
vq

Re: The elements of enlightenment

[Bart Schaefer]

On Mon, Dec 5, 2022 at 1:21 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> I have to keep appending to the list for each line, can the 'print -v
> var' method handle that?

     The -v option causes the output to be stored as the value of the
     parameter NAME, instead of printed.  If NAME is an array and the
     format string is reused when consuming arguments then one array
     element will be used for each use of the format string.

So
    typeset -a list
    printf -v list "%s ${red}%s${nrm}" ${(kv)options}
should do what you want.

Re: The elements of enlightenment

[Bart Schaefer]

On Mon, Dec 5, 2022 at 1:49 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
>     printf -v list "%s ${red}%s${nrm}" ${(kv)options}
> should do what you want.

Well, OK, not quite, that will color the "on" values red as well as
the "off", but you get the idea.

Re: The elements of enlightenment

[Ray Andrews]

On 2022-12-05 13:49, Bart Schaefer wrote:
> So
>      typeset -a list
>      printf -v list "%s ${red}%s${nrm}" ${(kv)options}
> should do what you want.

This seems to nail it:


    local k v
     local list=()
     local line=
     local -A color=(off ${red} on '')
     for k v in ${(kv)options}; do
         printf -v line '%-20s %s%s\e[0m' $k ${color[$v]} $v
         list+=$line
     done

... save a few miliseconds!   Very educational thanks gentlemen.

Re: The elements of enlightenment

[Bart Schaefer]

On Mon, Dec 5, 2022 at 2:24 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> This seems to nail it:

Try

local -a list
printf -v list "%-20s %s" ${(kv)options/#%off/${red}off$'\e[0m'}

No "for" loop needed, and saves a bunch of local declarations.

Re: The elements of enlightenment

[Ray Andrews]

On 2022-12-05 15:24, Bart Schaefer wrote:
> local -a list
> printf -v list "%-20s %s" ${(kv)options/#%off/${red}off$'\e[0m'}

Bloody marvelous, that's four levels of improvement just today! My 
backup system is overloaded. Amazing how much work you can do on one 
line.  And it's down to .03 seconds for 100 runs vs. a glacial .06 just 
previous.  So that'd be 200X faster than the original this morning -- 
which seems like a long time ago. Haven't learned so much in one day for 
a long time.

Re: The elements of enlightenment

[Lawrence Velázquez]

On Mon, Dec 5, 2022, at 10:57 AM, Ray Andrews wrote:
> By double quoting: ( "$( setopt )" ) we seem to have 
> solved the problem, it prints correctly.  But trouble lurks in the 
> shadows.   I do this:
>
>      $  list=( "${list[@]/ off/${red} off${nrm}}" )
>
> ... And I'm baffled that only the very first 'off' is colorized. Why?  
> Because '$#list' = 1!  It looks like we have each line as a separate 
> element but we don't.  It only looks that way because the newlines in 
> the output are still in there, they aren't 'print -l' newlines, they're 
> newlines in the data itself!! They look the same but they are *not* the 
> same.

As an aside, simply printing out data structures' contents is an
unreliable debugging tool, for reasons you have just discovered.

	% arr=($'a\nb\rc' $'d\re\nf')
	% print -l $arr
	a
	c
	e
	f

A better tool is "typeset -p", which outputs an accurate (albeit
sometimes difficult-to-read) representation.

	% typeset -p arr
	typeset -a arr=( $'a\nb\C-Mc' $'d\C-Me\nf' )

-- 
vq