zsh-users
 help / color / mirror / code / Atom feed
* why is eval needed?
@ 2022-11-19 14:34 Ray Andrews
  2022-11-19 14:43 ` Roman Perepelitsa
  2022-11-19 16:48 ` Stephane Chazelas
  0 siblings, 2 replies; 18+ messages in thread
From: Ray Andrews @ 2022-11-19 14:34 UTC (permalink / raw)
  To: Zsh Users

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

  In a script:

    local level='-L 2'
    tree $level
    #eval tree $level

run it:

     2 /aWorking/Zsh/Source/Wk 0 $ . testing
     tree: Missing argument to -L option.

If I use 'eval' it's fine.  Now, I roughly understand that we need 
'eval' when there's some complicated nesting of expansions or some other 
complexity, but I also recall that in almost every case where I've 
discussed an 'eval' here, I've been shown that it isn't really needed.  
In this case the expansion seems so trivial that I'm puzzled.  If 
'level' were some other switch that didn't require a space in it ('tree' 
demands a space between the 'L' and the number) then it works without 
eval, but what's the issue?  The '2' is there, why does 'tree' not see 
it?  Is this one of those invisible bogeys?  Something about argument 
counts or such?  But quoting didn't change the error message.



[-- Attachment #2: Type: text/html, Size: 1224 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-19 14:34 why is eval needed? Ray Andrews
@ 2022-11-19 14:43 ` Roman Perepelitsa
  2022-11-19 17:02   ` Ray Andrews
  2022-11-19 16:48 ` Stephane Chazelas
  1 sibling, 1 reply; 18+ messages in thread
From: Roman Perepelitsa @ 2022-11-19 14:43 UTC (permalink / raw)
  To: Ray Andrews; +Cc: Zsh Users

On Sat, Nov 19, 2022 at 3:35 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
>  In a script:
>
> local level='-L 2'
> tree $level

With the default options this this the same as this:

    tree '-L 2'

Is it clear why it doesn't work?

Roman.


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-19 14:34 why is eval needed? Ray Andrews
  2022-11-19 14:43 ` Roman Perepelitsa
@ 2022-11-19 16:48 ` Stephane Chazelas
  2022-11-19 19:12   ` Ray Andrews
  1 sibling, 1 reply; 18+ messages in thread
From: Stephane Chazelas @ 2022-11-19 16:48 UTC (permalink / raw)
  To: Ray Andrews; +Cc: Zsh Users

2022-11-19 06:34:30 -0800, Ray Andrews:
>  In a script:
> 
>    local level='-L 2'
>    tree $level
>    #eval tree $level
> 
> run it:
> 
>     2 /aWorking/Zsh/Source/Wk 0 $ . testing
>     tree: Missing argument to -L option.
> 
> If I use 'eval' it's fine.

We need eval when we need to evaluate some code stored as a
string dynamically. 

For instance, in perl, you could do:

$code_to_invoke_tree = 'system("tree", ';
$code_to_represent_arguments_for_tree = '"-L", "2"';
$code = $code_to_invoke_tree .
	$code_to_represent_arguments_for_tree . ');'
eval $code;

We have $code containing system("tree", "-L", "2");. That's perl
code which we evaluate with eval.

That looks very silly doesn't it?

Nobody's ever going to do that. Rather, you'd do:

$cmd = "tree";
@args_for_tree = ("-L", "2");
system($cmd, @args_for_tree);

It's exactly the same for shells except that the equivalent
syntax of perl's:

system("tree", "-L", "2");

to invoke tree with -L and 2 as arguments is:

tree -L 2

In shell, you separate arguments with whitespace instead of ",". You
don't need a system() function, because the shell's main job is
to execute commands that's the main thing it does, it would get
old quickly if had to use something like system() for everyone
of them. You don't need quotes around those arguments, because
everything is string in shells, since command arguments are
strings and that's the one thing shells deal with.

Now, you could also do:

shell_code_to_run_tree='tree '
shell_code_to_represent_arguments_for_tree='-L 2'
code=$shell_code_to_run_tree$shell_code_to_represent_arguments_for_tree
eval $code

but that would be equally silly. And like in perl you'd rather do:

cmd=tree
args_for_tree=(-L 2)
$cmd $args_for_tree

But you still need those -L and 2 to be two separate arguments,
hence the list variable.

If you do:

cmd=tree
arg='-L 2'
$cmd $arg

that's the same as:

$cmd = "tree";
$arg = '-L 2';
system($cmd, $arg);

And you're calling tree with only one "-L 2" argument while you
want to call it with 2 arguments: -L and 2.

-- 
Stephane




^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-19 14:43 ` Roman Perepelitsa
@ 2022-11-19 17:02   ` Ray Andrews
  2022-11-19 17:10     ` Roman Perepelitsa
  2022-11-19 18:02     ` Clinton Bunch
  0 siblings, 2 replies; 18+ messages in thread
From: Ray Andrews @ 2022-11-19 17:02 UTC (permalink / raw)
  To: zsh-users


On 2022-11-19 06:43, Roman Perepelitsa wrote:
> With the default options this this the same as this:
>      tree '-L 2'
>
> Is it clear why it doesn't work?

Actually yes ... but that was only my last effort.  With double quotes 
the error is:

 >  -L 2 [error opening dir]

... and with no quotes it's:

 > test:local:5: not valid in this context: -L

... to correct myself, it's really the double quoted offering that I'd 
expect to work.  I'm pretty sure this is an invisible grouping of 
arguments but I don't know what to do about it other than 'eval'.  I'm 
thinking that when tree sees: " -L 2 " it's seeing that not as a switch 
followed by a number, but as a single syntactic element which it's 
trying to interpret as a directory name.  Which sorta is 
semi-understandable since the quotes make " -L 2 " into 'one thing' 
whereas tree want's it to be 'two things'.  How to drop that variable in 
there so that tree is happy with it?  Is this perhaps a bit of 
crabbiness on the part of tree?  Should " -L 2 " be an array?? Sheesh, 
that might keep tree happy.  And ...

     local level=( -L 2 )
     tree $level

...

Success!  Ha! You give me enough feedback that I can talk myself out of 
the problem :-)







^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-19 17:02   ` Ray Andrews
@ 2022-11-19 17:10     ` Roman Perepelitsa
  2022-11-19 18:02     ` Clinton Bunch
  1 sibling, 0 replies; 18+ messages in thread
From: Roman Perepelitsa @ 2022-11-19 17:10 UTC (permalink / raw)
  To: Ray Andrews; +Cc: zsh-users

On Sat, Nov 19, 2022 at 6:03 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
 > test:local:5: not valid in this context: -L

The original code you've posted doesn't generate this error:

    % local level='-L 2'
    % tree $level
    tree: Missing argument to -L option.

This is as expected. The same as here:

    % tree '-L 2'
    tree: Missing argument to -L option.

I can get "not valid in this context" error with this:

    % local level -L
    local: not valid in this context: -L

You must be doing something differently compared to what you've posted.

Roman.


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-19 17:02   ` Ray Andrews
  2022-11-19 17:10     ` Roman Perepelitsa
@ 2022-11-19 18:02     ` Clinton Bunch
  2022-11-19 18:18       ` Roman Perepelitsa
  1 sibling, 1 reply; 18+ messages in thread
From: Clinton Bunch @ 2022-11-19 18:02 UTC (permalink / raw)
  To: zsh-users

On 11/19/2022 11:02 AM, Ray Andrews wrote:
>
> On 2022-11-19 06:43, Roman Perepelitsa wrote:
>> With the default options this this the same as this:
>>      tree '-L 2'
>>
>> Is it clear why it doesn't work?
>
> Actually yes ... but that was only my last effort.  With double quotes 
> the error is:
>
> >  -L 2 [error opening dir]
>
> ... and with no quotes it's:
>
> > test:local:5: not valid in this context: -L
>
> ... to correct myself, it's really the double quoted offering that I'd 
> expect to work.  I'm pretty sure this is an invisible grouping of 
> arguments but I don't know what to do about it other than 'eval'.  I'm 
> thinking that when tree sees: " -L 2 " it's seeing that not as a 
> switch followed by a number, but as a single syntactic element which 
> it's trying to interpret as a directory name.  Which sorta is 
> semi-understandable since the quotes make " -L 2 " into 'one thing' 
> whereas tree want's it to be 'two things'.  How to drop that variable 
> in there so that tree is happy with it?  Is this perhaps a bit of 
> crabbiness on the part of tree?  Should " -L 2 " be an array?? Sheesh, 
> that might keep tree happy.  And ...
>
>     local level=( -L 2 )
>     tree $level
>
> ...
>
> Success!  Ha! You give me enough feedback that I can talk myself out 
> of the problem :-)


Another option is

     local level='-L 2'

     tree $=level


Word splitting level gives the results you want

>
>
>
>
>
>



^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-19 18:02     ` Clinton Bunch
@ 2022-11-19 18:18       ` Roman Perepelitsa
  0 siblings, 0 replies; 18+ messages in thread
From: Roman Perepelitsa @ 2022-11-19 18:18 UTC (permalink / raw)
  To: Clinton Bunch; +Cc: zsh-users

On Sat, Nov 19, 2022 at 7:03 PM Clinton Bunch <cdb_zsh@zentaur.org> wrote:
>
> Another option is
>
>      local level='-L 2'
>
>      tree $=level

If you really have to use a scalar instead of an array and then split
it, then doing the following is usually the way to go:

    local args=(${(z)level})
    tree "${(@Q)args}"

This allows you to pass quoted and empty arguments:

    level='foo "bar baz" "" qux'

Note that if you combine the two expansions into one
("${(@Q)${(@z)level}}"), you'll have incorrect results when $level is
empty.

You can do the reverse transformation--from array to scalar--like this:

    local array=(...)
    local scalar=${(@q)array}

Roman.


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-19 16:48 ` Stephane Chazelas
@ 2022-11-19 19:12   ` Ray Andrews
  2022-11-19 19:50     ` Lawrence Velázquez
  0 siblings, 1 reply; 18+ messages in thread
From: Ray Andrews @ 2022-11-19 19:12 UTC (permalink / raw)
  To: zsh-users


On 2022-11-19 08:48, Stephane Chazelas wrote:
> And you're calling tree with only one "-L 2" argument while you
> want to call it with 2 arguments: -L and 2.

Right, that's what I figured out.  The hard part is 'seeing' these 
invisible boundaries.  But I'm pleased with myself that I did sort it out.

Clinton:

Another option is

     local level='-L 2'

     tree $=level


Word splitting level gives the results you want

     Excellent.  Yes, that's understandable.  Main thing is to realize 
that 'tree' want's two arguments there, not the intuitive single argument.

Roman:


You can do the reverse transformation--from array to scalar--like this:

     local array=(...)
     local scalar=${(@q)array}


... Finally I see a use of '(q)' that makes sense to me.



^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-19 19:12   ` Ray Andrews
@ 2022-11-19 19:50     ` Lawrence Velázquez
  2022-11-19 22:21       ` Ray Andrews
  0 siblings, 1 reply; 18+ messages in thread
From: Lawrence Velázquez @ 2022-11-19 19:50 UTC (permalink / raw)
  To: Ray Andrews; +Cc: zsh-users

On Sat, Nov 19, 2022, at 2:12 PM, Ray Andrews wrote:
>      Excellent.  Yes, that's understandable.  Main thing is to realize 
> that 'tree' want's two arguments there, not the intuitive single argument.

A "single argument" is not intuitive at all.  You would not run

    tree '-L 2'

or

    tree -L\ 2

so it does not make any sense to think that "tree" wants a single
argument there.

-- 
vq


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-19 19:50     ` Lawrence Velázquez
@ 2022-11-19 22:21       ` Ray Andrews
  2022-11-20  8:55         ` Stephane Chazelas
  0 siblings, 1 reply; 18+ messages in thread
From: Ray Andrews @ 2022-11-19 22:21 UTC (permalink / raw)
  To: zsh-users


On 2022-11-19 11:50, Lawrence Velázquez wrote:
>
> A "single argument" is not intuitive at all.  You would not run
>
>      tree '-L 2'
>
> or
>
>      tree -L\ 2
>
> so it does not make any sense to think that "tree" wants a single
> argument there.

Intuition is subjective.  Besides, usually such switches don't demand a 
space IIRC and it really is actually one argument, this is an exception. 
Even if the space is demanded, one might still think of a switch as a 
single semantic instruction even if it must be syntactically two words.  
A filename with a space in it is NOT broken in half, it's still one 
entity, so I'm thinking the same way.  I myself was naively thinking of 
it as nothing more than a string of four characters to be dropped into 
another string of characters -- as simple as that.  As if it was the 
command line. But commands do have their need to group characters into 
arguments so the invisible rules must be followed.  ' -L 2 ' must be 
invisibly broken in half.  Which, interestingly 'eval' seems to do 
automatically.  It's a huge thing coming to understand that what to me 
might look like 'just' a string of characters, to a command, needs to be 
viewed differently.  Once I understand that, things get much simpler.  I 
guess internally zsh must have all sorts of meta data attached to arrays 
to keep track of what's joined/split to/from what else.  It would be 
cool to be able to somehow see this invisible stuff, some way to see the 
invisible structure of a command line.  But I'll be ready the next time 
something like this crops up.





^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-19 22:21       ` Ray Andrews
@ 2022-11-20  8:55         ` Stephane Chazelas
  2022-11-20 13:47           ` Ray Andrews
  0 siblings, 1 reply; 18+ messages in thread
From: Stephane Chazelas @ 2022-11-20  8:55 UTC (permalink / raw)
  To: Ray Andrews; +Cc: zsh-users

2022-11-19 14:21:32 -0800, Ray Andrews:
[...]
> Intuition is subjective.  Besides, usually such switches don't demand a
> space IIRC and it really is actually one argument, this is an exception.
> Even if the space is demanded, one might still think of a switch as a single
> semantic instruction even if it must be syntactically two words.  A filename
> with a space in it is NOT broken in half, it's still one entity, so I'm
> thinking the same way.  I myself was naively thinking of it as nothing more
> than a string of four characters to be dropped into another string of
> characters -- as simple as that.  As if it was the command line. But
> commands do have their need to group characters into arguments so the
> invisible rules must be followed.  ' -L 2 ' must be invisibly broken in
> half.  Which, interestingly 'eval' seems to do automatically.  It's a huge
> thing coming to understand that what to me might look like 'just' a string
> of characters, to a command, needs to be viewed differently.  Once I
> understand that, things get much simpler.  I guess internally zsh must have
> all sorts of meta data attached to arrays to keep track of what's
> joined/split to/from what else.  It would be cool to be able to somehow see
> this invisible stuff, some way to see the invisible structure of a command
> line.  But I'll be ready the next time something like this crops up.
[...]

Just to clarify, when you enter:

tree -L 1

or:

eval 'tree -L 1'

/usr/bin/tree ends up being executed but it doesn't receive any
space character in any of the 3 arguments it receives. Not
anymore that it receives coma characters when you enter the
equivalent

system "tree","-L","1";

perl code. Those spaces are part of the shell language syntax, just
like the comas are part of the perl language syntax.

It's the caller's role to pass tree, -L and 1 as separate
arguments.

What tree sees is argv[0] = "tree", argv[1] = "-L", argv[2] =
"1".

tree parses its options. It looks like it does not use the
standard getopt() API or the GNU getopt_long() API for that:

$ nm -D =tree | grep -i getopt
$ ltrace -e '*opt*@*' tree > /dev/null
+++ exited (status 0) +++

So it must be doing it by hand.

If it were using getopt(), upon encountering "-L" in argv[1], it
would look for the value for that -L option after the L in
argv[1], and since here there's nothing, it would look for it in
argv[2], if there was no argv[2], getopt() would fail and an
error would be reported.

tree is not using getopt() and it looks like it only accepts
values for options that take values in the next argument.

But even if it used getopt(), in:

tree '-L 1'
tree $'-L\x201'
tree -L\ 1
arg='-L 1'; tree $arg


(or system("tree", "-L 1") in perl)

"tree" would be able to find a value for that -L option but that
value would be " 1", not "1", same as if you had called it with
-L ' 1' (or '--level= 1' or --level=' 1' or --level ' 1' if it
were using GNU getopt_long()) and depending on how it decodes
strings into numbers, it could very well choke on that leading
space.

As it happens, here it uses strtoul() to do that and doesn't
even seem to check that the input string is fully decoded, so
leading whitespace are ignored and it's very liberal as to what
it accepts:

$ ltrace -e '*strto*[lif]@*' tree -L $'\x20\t\r\n123.5e+20 whatever' > /dev/null
tree->strtoul(" \t\r\n123.5e+20 whatever", 0, 0)                                                  = 123
+++ exited (status 0) +++

(it got 123 out of that
<space><tab><cr><lf>123.5e+20<space>whatever argument.

In any case zsh (or perl) has no saying on how tree may parse
its options. You have to invoke it the way its meant to be
invoked and in the case of tree, the option arguments must be
passed as separate arguments.

For most other commands, you would be able to pass them either
as one -L1 argument or two -L and 1 arguments and also be able
to combine more than one option in a single argument as in
-xyL1 (assuming -x and -y are options that don't take
arguments, are flags/boolean).

-- 
Stephane


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-20  8:55         ` Stephane Chazelas
@ 2022-11-20 13:47           ` Ray Andrews
  2022-11-20 15:08             ` Stephane Chazelas
  0 siblings, 1 reply; 18+ messages in thread
From: Ray Andrews @ 2022-11-20 13:47 UTC (permalink / raw)
  To: zsh-users

On 2022-11-20 00:55, Stephane Chazelas wrote:


It's the caller's role to pass tree, -L and 1 as separate

> arguments.
Yeah I get it.  The first time I 'got it' was when discussing 'aptitude' 
and the various quotings needed.  Again one is creating an invisible 
level of organization imposed on a string of characters.  Where it got 
confusing was when zsh imposed it's own ideas of grouping, making '-L 2' 
into a single entity.  Dunno if it's really a thing to be desired but 
naively one might like some way of assembling a command string as if it 
was at CLI, that is *just* a string of characters -- which is how I was 
looking at it.
> tree parses its options. It looks like it does not use the
> standard getopt() API or the GNU getopt_long() API for that:
>
> $ nm -D =tree | grep -i getopt
> $ ltrace -e '*opt*@*' tree > /dev/null
> +++ exited (status 0) +++
What are you doing there?  I have no 'nm' command here.  No such command 
in Debian repository.
>
> So it must be doing it by hand.
Yeah, so much in the GNU/Linux world is ad hoc.  Everybody did their own 
thing.  No rules.
> In any case zsh (or perl) has no saying on how tree may parse
> its options. You have to invoke it the way its meant to be
> invoked and in the case of tree, the option arguments must be
> passed as separate arguments.
Sure.  The main thing is to just understand that.  Not too difficult 
when you do, something might need splitting or joining but it won't take 
long to figure out.  But why does calling 'eval' fix it?  It seems as if 
eval 'flattens' everything -- one is back to a command line string of 
characters with no imposed grouping.
>
> For most other commands, you would be able to pass them either
> as one -L1 argument or two -L and 1 arguments and also be able
> to combine more than one option in a single argument as in
> -xyL1 (assuming -x and -y are options that don't take
> arguments, are flags/boolean).
>
Yeah, as I was saying, it does seem that 'tree' is very crabby. Why 
don't the GNU people iron these things out?




^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-20 13:47           ` Ray Andrews
@ 2022-11-20 15:08             ` Stephane Chazelas
  2022-11-20 16:27               ` Ray Andrews
  0 siblings, 1 reply; 18+ messages in thread
From: Stephane Chazelas @ 2022-11-20 15:08 UTC (permalink / raw)
  To: Ray Andrews; +Cc: zsh-users

2022-11-20 05:47:33 -0800, Ray Andrews:
[...]
> > arguments.
> Yeah I get it.  The first time I 'got it' was when discussing 'aptitude' and
> the various quotings needed.  Again one is creating an invisible level of
> organization imposed on a string of characters.  Where it got confusing was
> when zsh imposed it's own ideas of grouping, making '-L 2' into a single
> entity.

zsh doesn't impose anything. "-L 2" is a string made of 4
characters in any shell or programming language. There's no
programming language where "-L 2" means 2 strings -L and 2.

What may be the source of your confusion is that in the Bourne
shell, there was a second round of splitting applied to unquoted
text in list context. Space is a special character in the syntax
of the shell which is used to delimit command arguments, but
it's also in the default value of the $IFS special variable
which was used to *further* split text into futher arguments.

In the Bourne shell (introduced in the late 70s):

edit file

Is like in all other shells code that's meant to run
/path/to/edit with "edit" and "file" as separate arguments, but
if $IFS also happens to contain "i", then those two words happen
to be further  split into "ed", "t" and "f", "le", so you end up
running "/bin/ed" with "ed", "t", "f" and "le" as arguments.

That splitting also happened on top of parameter expansion.

ed $file

Upon syntax evaluation yields "ed" and "$file" as two separate
tokens, but both "ed" and the contents of $file further
underwent $IFS-splitting. Almost worse, globbing also happened
on top of $file expansion.

That is a very weird feature from a language design point of
view. Anybody who doesn't know what the shells looked like
before the Bourne shell would think Stephen Bourne was out of
his mind.

The thing is in the original Unix shell (designed in the early
70s on computers than had kilobytes of RAM), there was no
variable. There was a concept of script that could take arguments,
and in the script, you'd refer to them as $1, $2... the
positional parameters (which have survived in modern shells). 

Parameter expansion was really crude, it was a bit like aliases,
the contents of the parameter was just expanded in place into
the code being evaluated.

So for instance, if you called your script as:

my-script 'hi;rm -rf /'

And the script did:

echo $1

That would say hi and destroy the system.

It had its uses though.

my-script 'dir1 dir2' 'file3 file4'

With a script that did:

ls $1
rm $2

Would list the dir1 and dir2 directories and remove  the file3
and file4 files.

And:

my-script '*.txt'

in a script that did:

cd /foo
rm $1
cd /bar
rm $1

Would remove the files with name ending in .txt in both /foo and
/bar (or the current directory  if cd failed... but that's
beside the point).

My understanding is that the Bourne shell's bizarre IFS handling
and the fact that globbing was performed upon parameter
expansion was an attempt to keep some level of backward
compatibility with that Thompson shell. You see similar things
hapenning in csh from the same era.

The Korn shell (from the early 80s) kept most of that with the
exception that it only did IFS-splitting upon expansions, not on
literal text. 

In:

file=/some/file
edit $file

With IFS=i, $file would be split  into /some/f and le, but edit
would stay edit.

The POSIX specification of sh standardised that behaviour so
that's the one found in bash / dash / yash... as well.

That nonsense was fixed in most shells written after that
though, starting with rc, the shell of plan9 / Unix v10 from the
late 80s, but also zsh (early 90s)  or more recently (mid 2000s)
fish.

The fact that in sh/ksh/bash:

arg='-L 1'
tree $arg

Calls tree with "-L" and "1" as arguments is not that bash
doesn't do some sort of magic "grouping" that zsh would be
doing, but that ksh/bash contrary to zsh does that extra layer
of $IFS-splitting from the Bourne shell on top of the syntax
parsing as the default value of $IFS happens to contain the
space character.

That's why in sh/bash/ksh you almost always need to quote
parameter expansions if you intend to pass the contents of a
variable as an argument to a command.

See
https://unix.stackexchange.com/questions/171346/security-implications-of-forgetting-to-quote-a-variable-in-bash-posix-shells
for the kind of thing that  can happen if you forget.


> Dunno if it's really a thing to be desired but naively one might
> like some way of assembling a command string as if it was at CLI, that is
> *just* a string of characters -- which is how I was looking at it.

If you want to assemble a string, and that string to be
evaluated as shell code, that's what eval is for. eval evaluate
code written in the shell language. That's a way  to dynamically
invoke the language interpreter.

But generally, that's not what you want. Writing correct shell
code dynamically based on external input is very easy to get
wrong. In your case, it rather looks like you want to build up a
list of arguments to pass to a command for which you obviously
need a shell list/array variable.

> > tree parses its options. It looks like it does not use the
> > standard getopt() API or the GNU getopt_long() API for that:
> > 
> > $ nm -D =tree | grep -i getopt
> > $ ltrace -e '*opt*@*' tree > /dev/null
> > +++ exited (status 0) +++
> What are you doing there?  I have no 'nm' command here.  No such command in
> Debian repository.

nm is a standard development command (though not -D which AFAIK
is a GNU extension). Part of GNU binutils on GNU systems. Here
used to list the external functions that the utility claims it
needs.

The above shows that tree doesn't use the getopt() standard
command or that if it does, it embedded a copy into the
executable rather than using the one from the GNU libc.

> > So it must be doing it by hand.
> Yeah, so much in the GNU/Linux world is ad hoc.  Everybody did their own
> thing.  No rules.

Note that tree is not part of the GNU project, it's just a
utility written by some guy and shared to the world. There is
*some* level of consistency among utilities in the GNU
toolchest. There is even such a thing as published GNU coding standards.
See for instance
https://www.gnu.org/prep/standards/html_node/Command_002dLine-Interfaces.html#Command_002dLine-Interfaces

zsh is not part of the GNU project either. Note that zsh also
predates Linux and has been used in a wide variety of GNU and
non-GNU systems.

I'll agree with you that the lack of consistency in API between the
different tools that are available out there can be annoying
(nothing to do with GNU or Linux) 

[...]
> figure out.  But why does calling 'eval' fix it?  It seems as if eval
> 'flattens' everything -- one is back to a command line string of characters
> with no imposed grouping.

eval evaluates shell code. I struggle to understand what you
don't understand. Maybe you're thinking too much or too little
of what a "command line string" is. A "command line string"
like:

var='value'; if blah; then echo $(cmd) | tr -d x; fi

is just one line of code in the syntax of the shell programing
language. It's not  some magic universal language to talk to the
system.

You can store that code in bits in variables with:

a="'val"
b="ue'; if bl"
c='ah; then ech'
d='o $(cmd) | tr -'
e='d x; fi'

But surely you don't expact

var=$a$b$c$d$e

to have the same effect as running that command.

You can however do:

eval "var=$a$b$c$d$e"

For that string to be passed as code to the shell language
interpreter for it to interpret.

[...]
> Yeah, as I was saying, it does seem that 'tree' is very crabby. Why don't
> the GNU people iron these things out?
[...]

Again, tree has nothing to do with the GNU project. There is
also unfortunatly not one standard to parse options and
arguments. There is POSIX getopt() but for instance, it doesn't
support long options (neither a la GNU nor a la X11, or a la
perl Getopt::Long nor a la zsh zparseopts...).

-- 
Stephane


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-20 15:08             ` Stephane Chazelas
@ 2022-11-20 16:27               ` Ray Andrews
  2022-11-20 20:16                 ` Stephane Chazelas
  0 siblings, 1 reply; 18+ messages in thread
From: Ray Andrews @ 2022-11-20 16:27 UTC (permalink / raw)
  To: zsh-users


On 2022-11-20 07:08, Stephane Chazelas wrote:
> zsh doesn't impose anything. "-L 2" is a string made of 4
> characters in any shell or programming language. There's no
> programming language where "-L 2" means 2 strings -L and 2.

But that's the thing, I'd naively thought the variable could be inserted 
in the command string and be 'just' a string of characters, but zsh 
imposes that 'tree' must see '-L 2' as a single entity, yes?  As shown, 
we need word splitting to solve the problem and show tree what it wants 
to see. Or, as I was speculating, some way of just flattening the string 
back to nothing more than a string of characters -- but then again, 
probably tree wouldn't like that either, perhaps the string is always 
'packaged' into words?  If so, then there's no avoiding that we must 
word-split '-L 2' into two words and there's nothing to wish for.


> That is a very weird feature from a language design point of
> view. Anybody who doesn't know what the shells looked like
> before the Bourne shell would think Stephen Bourne was out of
> his mind.
>
It's what happens when functionality expands by accretion -- simple 
structures no longer suffice but they are tweaked rather than 
rethought.  It's like the difference between the layout of Paris vs. 
London.
> My understanding is that the Bourne shell's bizarre IFS handling
> and the fact that globbing was performed upon parameter
> expansion was an attempt to keep some level of backward
> compatibility with that Thompson shell. You see similar things
> hapenning in csh from the same era.
The dilemma of compatibility!  No easy answers.  I myself tend towards a 
'garage to the dump' mentality but it's not that simple.
> The Korn shell (from the early 80s) kept most of that with the
> exception that it only did IFS-splitting upon expansions, not on
> literal text.
Holy cow!  IFS splitting in text??
> The fact that in sh/ksh/bash:
>
> arg='-L 1'
> tree $arg
>
> Calls tree with "-L" and "1" as arguments is not that bash
> doesn't do some sort of magic "grouping" that zsh would be
> doing, but that ksh/bash contrary to zsh does that extra layer
> of $IFS-splitting from the Bourne shell on top of the syntax
> parsing as the default value of $IFS happens to contain the
> space character.
Ah!  So my little issue is a zsh exclusive?  Not complaining tho, I 
don't like zsh doing things I didn't ask it to do so if in this case I 
need to request a word split that's just fine.  After all it's tree 
that's being difficult so the problem is rare and easily solved.
> That's why in sh/bash/ksh you almost always need to quote
> parameter expansions if you intend to pass the contents of a
> variable as an argument to a command.
>
> See
> https://unix.stackexchange.com/questions/171346/security-implications-of-forgetting-to-quote-a-variable-in-bash-posix-shells
> for the kind of thing that  can happen if you forget.
Heavy.  That's the closest I've come to understanding those vulnerabilities.
>
> If you want to assemble a string, and that string to be
> evaluated as shell code, that's what eval is for. eval evaluate
> code written in the shell language. That's a way  to dynamically
> invoke the language interpreter.
Ok, so I understood that correctly.  At least partially.
> What are you doing there?  I have no 'nm' command here.  No such command in
> Debian repository.
> nm is a standard development command (though not -D which AFAIK
> is a GNU extension). Part of GNU binutils on GNU systems.
Ah, that's why I can't find it.  It's hard to find commands so packaged.
> Note that tree is not part of the GNU project, it's just a
> utility written by some guy and shared to the world. There is
> *some* level of consistency among utilities in the GNU
> toolchest. There is even such a thing as published GNU coding standards.
> See for instance
> https://www.gnu.org/prep/standards/html_node/Command_002dLine-Interfaces.html#Command_002dLine-Interfaces
Yeah, tides of organization wash here and there.  No overall commander 
tho, so a bit of chaos is unavoidable.
> I'll agree with you that the lack of consistency in API between the
> different tools that are available out there can be annoying
> (nothing to do with GNU or Linux)
You know, half of it is just cultural adaptation.  When I first started 
with Linux the first thing I did was decide between zsh and bash and I'm 
basically assuming that Linux will be like DOS but slightly more 
powerful and zsh will be like C and that all Linux OS apps will be as 
flawless as WordPerfect 5.1.  CRASH!  Now that I'm used to being in a 
bazaar not a monastery it's easier to get around.  One does get mud on 
one's shoes.
> eval evaluates shell code. I struggle to understand what you
> don't understand. Maybe you're thinking too much or too little
> of what a "command line string" is. A "command line string"
> like:

I think at this level of my education it's enough to say that eval looks 
at it's arguments as if they were typed on the command line.  I haven't 
really 'understood' anything, I use eval ad hoc when it seems to solve 
problems.  Using it with real understanding is a future goal.


> For that string to be passed as code to the shell language
> interpreter for it to interpret.

Right, I get that:


2 /aWorking/Zsh/Source/Wk 0 $ var="echo howdy"

2 /aWorking/Zsh/Source/Wk 0 $ eval $var
howdy

> Again, tree has nothing to do with the GNU project.

I dunno, I get it all from Debian and Debian is a GNU/Linux OS so I 
don't know more than that.


Thanks for a most educational post!  I love these history lessons, they 
inform my entire outlook.  You can't understand where you are unless you 
understand where you've been.




^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-20 16:27               ` Ray Andrews
@ 2022-11-20 20:16                 ` Stephane Chazelas
  2022-11-20 22:31                   ` Bart Schaefer
  2022-11-20 22:47                   ` Ray Andrews
  0 siblings, 2 replies; 18+ messages in thread
From: Stephane Chazelas @ 2022-11-20 20:16 UTC (permalink / raw)
  To: Ray Andrews; +Cc: zsh-users

2022-11-20 08:27:01 -0800, Ray Andrews:
> 
> On 2022-11-20 07:08, Stephane Chazelas wrote:
> > zsh doesn't impose anything. "-L 2" is a string made of 4
> > characters in any shell or programming language. There's no
> > programming language where "-L 2" means 2 strings -L and 2.
> 
> But that's the thing, I'd naively thought the variable could be inserted in
> the command string and be 'just' a string of characters, but zsh imposes
> that 'tree' must see '-L 2' as a single entity, yes?

No, you have it completely backwards. And I think at the heart
of your misunderstanding there is the misconception that you
pass a "command string" to an executable.

You do not, you pass a list of arguments.

On Unix-like systems, execution of commands are with the execve()
system call which takes 3 arguments:

1. the path name to the executable
2. a list of arguments: argv[]
3. a list of environment variables: envp[]

What is commonly refered to as a command line is the shell's
interface to that system call.

VAR1=foo VAR2=bar cmd 'arg 1' arg2

Is how you tell your shell to do a

execve("/path/to/cmd", ["cmd", "arg 1", "arg2"],
[exportedvariable, "VAR1=foo", "VAR2=bar"])


You do not pass the "cmd 'arg 1' arg2" string as argument to
/path/to/cmd.

>  As shown, we need word
> splitting to solve the problem and show tree what it wants to see. Or, as I
> was speculating, some way of just flattening the string back to nothing more
> than a string of characters -- but then again, probably tree wouldn't like
> that either, perhaps the string is always 'packaged' into words?  If so,
> then there's no avoiding that we must word-split '-L 2' into two words and
> there's nothing to wish for.

Why store those two arguments as a string with a space in between them
and then ask the shell to split it, that makes no sense.

Just store those two arguments and pass them to tree, that's exactly
what arrays are for. In several shells like rc or fish, all variables
are arrays (also in csh, but csh has many other issues of its own).
From a design point of view, that's the most sensible thing to do as
what shells deal with primarily is the list of arguments to commands¹

zsh: args=(-L 1); tree $args
rc: args=(-L 1); tree $args
fish: set args -L 1; tree $args
tcsh: set args = (-L 1); tree $args:q
ksh: set -A args -- -L 1; tree "${args[@]}"
ksh93/bash2+(/zsh): args=(-L 1); tree "${args[@]}"

(¹ a complication and a spanner in the works is that environment
variables (those VAR=value strings passed in envp[] above) are
string/scalar only; in csh and in the Bourne shell, shell variables and
env variables were more separated than they are in modern shells).

If you wish

arg='-L 2'
tree $arg

passed "-L" and "2" as separate arguments to tree, do you also wish
that:

file='my file.txt'
rm -- $file

Also passed "my" and "file.txt" as separate arguments to rm for rm to
unlink those 2 files?

What about:

files=(
  'my file.txt'
  'my other file.txt'
)
rm -- $files

Should they be split too?

How about:

text='shutdown|reboot'
echo $text

Should that run echo shutdown|reboot and reboot?


[...]
> > Calls tree with "-L" and "1" as arguments is not that bash
> > doesn't do some sort of magic "grouping" that zsh would be
> > doing, but that ksh/bash contrary to zsh does that extra layer
> > of $IFS-splitting from the Bourne shell on top of the syntax
> > parsing as the default value of $IFS happens to contain the
> > space character.
> Ah!  So my little issue is a zsh exclusive?

No, again, any recent shell (i.e. post 1980) that had no intention of
being compatible with the Bourne shell including rc (and derivatives
like akanga or es) and fish behave like zsh in this instance².

In fish,

set arg '-L 1'
tree $arg

Or in rc/es/akanga:

arg = '-L 1'
tree $arg

all call tree with one "-L 1" argument.

(² and even better, zsh still has that issue that unquoted expansions to
empty removal, one of the issues with Bourne-like shells that it hasn't
fixed. rc/es/akanga/fish have).

>  Not complaining tho, I don't
> like zsh doing things I didn't ask it to do so if in this case I need to
> request a word split that's just fine.  After all it's tree that's being
> difficult so the problem is rare and easily solved.

No, it's not tree being difficult. In any command doing

cmd '-o arg'

is passing the " arg" string as argument to the -o option.

sort '-o file' file

Will sort the "file" file into the " file" file just like you asked. If
you want to sort it into itself, it's sort -o file file

[...]
> > nm is a standard development command (though not -D which AFAIK
> > is a GNU extension). Part of GNU binutils on GNU systems.
> Ah, that's why I can't find it.  It's hard to find commands so packaged.

apt-file is your friend.

For the POSIX standard (2018 edition) see
https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/

[...]
> > Again, tree has nothing to do with the GNU project.
> 
> I dunno, I get it all from Debian and Debian is a GNU/Linux OS so I don't
> know more than that.

On Debian

dpkg -S =tree

Will tell you what package the tree command comes from.

apt showsrc that-package

Will tell you how it was built and generally where the source came from.
Here, not from gnu.org but from http://mama.indstate.edu/users/ice/tree/
http://mama.indstate.edu/users/ice/tree/changes.html shows it has a long
history and might be even as old as zsh.

-- 
Stephane


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-20 20:16                 ` Stephane Chazelas
@ 2022-11-20 22:31                   ` Bart Schaefer
  2022-11-21  2:13                     ` Ray Andrews
  2022-11-20 22:47                   ` Ray Andrews
  1 sibling, 1 reply; 18+ messages in thread
From: Bart Schaefer @ 2022-11-20 22:31 UTC (permalink / raw)
  To: Ray Andrews, zsh-users

On Sun, Nov 20, 2022 at 12:17 PM Stephane Chazelas
<stephane@chazelas.org> wrote:
>
> No, you have it completely backwards. And I think at the heart
> of your misunderstanding there is the misconception that you
> pass a "command string" to an executable.

This misconception may be an effect of Ray's background with
DOS/Windows command interpreters.

In that world, each command is responsible both for parsing its
"arguments" and for interpreting them.  There was (is, I suppose) a
standard DLL to support the parsing part, and a lot of commands used
it, but it was still down to the command itself to call that DLL.

In the Unix-based world, it's expected that the shell will do the
parsing and the command only has to do the interpreting, so there are
a lot more rules about the structure of the command line.


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-20 20:16                 ` Stephane Chazelas
  2022-11-20 22:31                   ` Bart Schaefer
@ 2022-11-20 22:47                   ` Ray Andrews
  1 sibling, 0 replies; 18+ messages in thread
From: Ray Andrews @ 2022-11-20 22:47 UTC (permalink / raw)
  To: zsh-users


On 2022-11-20 12:16, Stephane Chazelas wrote:
>
>> But that's the thing, I'd naively thought the variable could be inserted in
>> the command string and be 'just' a string of characters, but zsh imposes
>> that 'tree' must see '-L 2' as a single entity, yes?
> No, you have it completely backwards.
But everything so far seems to tell me that I'm right.  When '-L 2' is 
presented as two arguments instead of one, everything is fine. Are we 
miscommunicating?
> And I think at the heart
> of your misunderstanding there is the misconception that you
> pass a "command string" to an executable.
>
> You do not, you pass a list of arguments.

> I know, but at least on the command line one can think of it as a string.  I understand that it it will be parsed as 'words' (including spaces if quoted).
>
> Why store those two arguments as a string with a space in between them
> and then ask the shell to split it, that makes no sense.
I know that now.  But as we discussed, the normal thing is for a switch 
to present as a single argument so that's what I did.  I quite 
understand the mistake NOW.
> (¹ a complication and a spanner in the works is that environment
> variables (those VAR=value strings passed in envp[] above) are
> string/scalar only; in csh and in the Bourne shell, shell variables and
> env variables were more separated than they are in modern shells).
Awesome that you can keep all this stuff in memory and not get your 
wires crossed.
>
> What about:
>
> files=(
>    'my file.txt'
>    'my other file.txt'
> )
> rm -- $files
>
> Should they be split too?
No, I understand that well.  I think this is a closed issue, I think I'm 
quite content.
> Ah!  So my little issue is a zsh exclusive?
> No, again, any recent shell
Ok, not 'exclusive' but it's been a point of difference between at least 
zsh and bash.
> No, it's not tree being difficult. In any command doing 

Bbbbut we've agreed that tree is being difficult!  You've explained it 
in great detail.

> apt-file is your friend.
So it is!
> Will tell you what package the tree command comes from.
>
> apt showsrc that-package

It's just not something I need to worry about.  To me it's all 'Linux'.  
Maybe one day I'll download some sources (if in C) and start tinkering.  
For now it just doesn't matter.   The thing is to get a bit more 
competent with zsh.





^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: why is eval needed?
  2022-11-20 22:31                   ` Bart Schaefer
@ 2022-11-21  2:13                     ` Ray Andrews
  0 siblings, 0 replies; 18+ messages in thread
From: Ray Andrews @ 2022-11-21  2:13 UTC (permalink / raw)
  To: zsh-users


> In the Unix-based world, it's expected that the shell will do the
> parsing and the command only has to do the interpreting, so there are
> a lot more rules about the structure of the command line.

Hit the nail on the head :-)




^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2022-11-21  2:16 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-19 14:34 why is eval needed? Ray Andrews
2022-11-19 14:43 ` Roman Perepelitsa
2022-11-19 17:02   ` Ray Andrews
2022-11-19 17:10     ` Roman Perepelitsa
2022-11-19 18:02     ` Clinton Bunch
2022-11-19 18:18       ` Roman Perepelitsa
2022-11-19 16:48 ` Stephane Chazelas
2022-11-19 19:12   ` Ray Andrews
2022-11-19 19:50     ` Lawrence Velázquez
2022-11-19 22:21       ` Ray Andrews
2022-11-20  8:55         ` Stephane Chazelas
2022-11-20 13:47           ` Ray Andrews
2022-11-20 15:08             ` Stephane Chazelas
2022-11-20 16:27               ` Ray Andrews
2022-11-20 20:16                 ` Stephane Chazelas
2022-11-20 22:31                   ` Bart Schaefer
2022-11-21  2:13                     ` Ray Andrews
2022-11-20 22:47                   ` Ray Andrews

Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/zsh/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).