* why is eval needed? @ 2022-11-19 14:34 Ray Andrews 2022-11-19 14:43 ` Roman Perepelitsa 2022-11-19 16:48 ` Stephane Chazelas 0 siblings, 2 replies; 18+ messages in thread From: Ray Andrews @ 2022-11-19 14:34 UTC (permalink / raw) To: Zsh Users [-- Attachment #1: Type: text/plain, Size: 891 bytes --] In a script: local level='-L 2' tree $level #eval tree $level run it: 2 /aWorking/Zsh/Source/Wk 0 $ . testing tree: Missing argument to -L option. If I use 'eval' it's fine. Now, I roughly understand that we need 'eval' when there's some complicated nesting of expansions or some other complexity, but I also recall that in almost every case where I've discussed an 'eval' here, I've been shown that it isn't really needed. In this case the expansion seems so trivial that I'm puzzled. If 'level' were some other switch that didn't require a space in it ('tree' demands a space between the 'L' and the number) then it works without eval, but what's the issue? The '2' is there, why does 'tree' not see it? Is this one of those invisible bogeys? Something about argument counts or such? But quoting didn't change the error message. [-- Attachment #2: Type: text/html, Size: 1224 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-19 14:34 why is eval needed? Ray Andrews @ 2022-11-19 14:43 ` Roman Perepelitsa 2022-11-19 17:02 ` Ray Andrews 2022-11-19 16:48 ` Stephane Chazelas 1 sibling, 1 reply; 18+ messages in thread From: Roman Perepelitsa @ 2022-11-19 14:43 UTC (permalink / raw) To: Ray Andrews; +Cc: Zsh Users On Sat, Nov 19, 2022 at 3:35 PM Ray Andrews <rayandrews@eastlink.ca> wrote: > > In a script: > > local level='-L 2' > tree $level With the default options this this the same as this: tree '-L 2' Is it clear why it doesn't work? Roman. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-19 14:43 ` Roman Perepelitsa @ 2022-11-19 17:02 ` Ray Andrews 2022-11-19 17:10 ` Roman Perepelitsa 2022-11-19 18:02 ` Clinton Bunch 0 siblings, 2 replies; 18+ messages in thread From: Ray Andrews @ 2022-11-19 17:02 UTC (permalink / raw) To: zsh-users On 2022-11-19 06:43, Roman Perepelitsa wrote: > With the default options this this the same as this: > tree '-L 2' > > Is it clear why it doesn't work? Actually yes ... but that was only my last effort. With double quotes the error is: > -L 2 [error opening dir] ... and with no quotes it's: > test:local:5: not valid in this context: -L ... to correct myself, it's really the double quoted offering that I'd expect to work. I'm pretty sure this is an invisible grouping of arguments but I don't know what to do about it other than 'eval'. I'm thinking that when tree sees: " -L 2 " it's seeing that not as a switch followed by a number, but as a single syntactic element which it's trying to interpret as a directory name. Which sorta is semi-understandable since the quotes make " -L 2 " into 'one thing' whereas tree want's it to be 'two things'. How to drop that variable in there so that tree is happy with it? Is this perhaps a bit of crabbiness on the part of tree? Should " -L 2 " be an array?? Sheesh, that might keep tree happy. And ... local level=( -L 2 ) tree $level ... Success! Ha! You give me enough feedback that I can talk myself out of the problem :-) ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-19 17:02 ` Ray Andrews @ 2022-11-19 17:10 ` Roman Perepelitsa 2022-11-19 18:02 ` Clinton Bunch 1 sibling, 0 replies; 18+ messages in thread From: Roman Perepelitsa @ 2022-11-19 17:10 UTC (permalink / raw) To: Ray Andrews; +Cc: zsh-users On Sat, Nov 19, 2022 at 6:03 PM Ray Andrews <rayandrews@eastlink.ca> wrote: > > test:local:5: not valid in this context: -L The original code you've posted doesn't generate this error: % local level='-L 2' % tree $level tree: Missing argument to -L option. This is as expected. The same as here: % tree '-L 2' tree: Missing argument to -L option. I can get "not valid in this context" error with this: % local level -L local: not valid in this context: -L You must be doing something differently compared to what you've posted. Roman. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-19 17:02 ` Ray Andrews 2022-11-19 17:10 ` Roman Perepelitsa @ 2022-11-19 18:02 ` Clinton Bunch 2022-11-19 18:18 ` Roman Perepelitsa 1 sibling, 1 reply; 18+ messages in thread From: Clinton Bunch @ 2022-11-19 18:02 UTC (permalink / raw) To: zsh-users On 11/19/2022 11:02 AM, Ray Andrews wrote: > > On 2022-11-19 06:43, Roman Perepelitsa wrote: >> With the default options this this the same as this: >> tree '-L 2' >> >> Is it clear why it doesn't work? > > Actually yes ... but that was only my last effort. With double quotes > the error is: > > > -L 2 [error opening dir] > > ... and with no quotes it's: > > > test:local:5: not valid in this context: -L > > ... to correct myself, it's really the double quoted offering that I'd > expect to work. I'm pretty sure this is an invisible grouping of > arguments but I don't know what to do about it other than 'eval'. I'm > thinking that when tree sees: " -L 2 " it's seeing that not as a > switch followed by a number, but as a single syntactic element which > it's trying to interpret as a directory name. Which sorta is > semi-understandable since the quotes make " -L 2 " into 'one thing' > whereas tree want's it to be 'two things'. How to drop that variable > in there so that tree is happy with it? Is this perhaps a bit of > crabbiness on the part of tree? Should " -L 2 " be an array?? Sheesh, > that might keep tree happy. And ... > > local level=( -L 2 ) > tree $level > > ... > > Success! Ha! You give me enough feedback that I can talk myself out > of the problem :-) Another option is local level='-L 2' tree $=level Word splitting level gives the results you want > > > > > > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-19 18:02 ` Clinton Bunch @ 2022-11-19 18:18 ` Roman Perepelitsa 0 siblings, 0 replies; 18+ messages in thread From: Roman Perepelitsa @ 2022-11-19 18:18 UTC (permalink / raw) To: Clinton Bunch; +Cc: zsh-users On Sat, Nov 19, 2022 at 7:03 PM Clinton Bunch <cdb_zsh@zentaur.org> wrote: > > Another option is > > local level='-L 2' > > tree $=level If you really have to use a scalar instead of an array and then split it, then doing the following is usually the way to go: local args=(${(z)level}) tree "${(@Q)args}" This allows you to pass quoted and empty arguments: level='foo "bar baz" "" qux' Note that if you combine the two expansions into one ("${(@Q)${(@z)level}}"), you'll have incorrect results when $level is empty. You can do the reverse transformation--from array to scalar--like this: local array=(...) local scalar=${(@q)array} Roman. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-19 14:34 why is eval needed? Ray Andrews 2022-11-19 14:43 ` Roman Perepelitsa @ 2022-11-19 16:48 ` Stephane Chazelas 2022-11-19 19:12 ` Ray Andrews 1 sibling, 1 reply; 18+ messages in thread From: Stephane Chazelas @ 2022-11-19 16:48 UTC (permalink / raw) To: Ray Andrews; +Cc: Zsh Users 2022-11-19 06:34:30 -0800, Ray Andrews: > In a script: > > local level='-L 2' > tree $level > #eval tree $level > > run it: > > 2 /aWorking/Zsh/Source/Wk 0 $ . testing > tree: Missing argument to -L option. > > If I use 'eval' it's fine. We need eval when we need to evaluate some code stored as a string dynamically. For instance, in perl, you could do: $code_to_invoke_tree = 'system("tree", '; $code_to_represent_arguments_for_tree = '"-L", "2"'; $code = $code_to_invoke_tree . $code_to_represent_arguments_for_tree . ');' eval $code; We have $code containing system("tree", "-L", "2");. That's perl code which we evaluate with eval. That looks very silly doesn't it? Nobody's ever going to do that. Rather, you'd do: $cmd = "tree"; @args_for_tree = ("-L", "2"); system($cmd, @args_for_tree); It's exactly the same for shells except that the equivalent syntax of perl's: system("tree", "-L", "2"); to invoke tree with -L and 2 as arguments is: tree -L 2 In shell, you separate arguments with whitespace instead of ",". You don't need a system() function, because the shell's main job is to execute commands that's the main thing it does, it would get old quickly if had to use something like system() for everyone of them. You don't need quotes around those arguments, because everything is string in shells, since command arguments are strings and that's the one thing shells deal with. Now, you could also do: shell_code_to_run_tree='tree ' shell_code_to_represent_arguments_for_tree='-L 2' code=$shell_code_to_run_tree$shell_code_to_represent_arguments_for_tree eval $code but that would be equally silly. And like in perl you'd rather do: cmd=tree args_for_tree=(-L 2) $cmd $args_for_tree But you still need those -L and 2 to be two separate arguments, hence the list variable. If you do: cmd=tree arg='-L 2' $cmd $arg that's the same as: $cmd = "tree"; $arg = '-L 2'; system($cmd, $arg); And you're calling tree with only one "-L 2" argument while you want to call it with 2 arguments: -L and 2. -- Stephane ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-19 16:48 ` Stephane Chazelas @ 2022-11-19 19:12 ` Ray Andrews 2022-11-19 19:50 ` Lawrence Velázquez 0 siblings, 1 reply; 18+ messages in thread From: Ray Andrews @ 2022-11-19 19:12 UTC (permalink / raw) To: zsh-users On 2022-11-19 08:48, Stephane Chazelas wrote: > And you're calling tree with only one "-L 2" argument while you > want to call it with 2 arguments: -L and 2. Right, that's what I figured out. The hard part is 'seeing' these invisible boundaries. But I'm pleased with myself that I did sort it out. Clinton: Another option is local level='-L 2' tree $=level Word splitting level gives the results you want Excellent. Yes, that's understandable. Main thing is to realize that 'tree' want's two arguments there, not the intuitive single argument. Roman: You can do the reverse transformation--from array to scalar--like this: local array=(...) local scalar=${(@q)array} ... Finally I see a use of '(q)' that makes sense to me. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-19 19:12 ` Ray Andrews @ 2022-11-19 19:50 ` Lawrence Velázquez 2022-11-19 22:21 ` Ray Andrews 0 siblings, 1 reply; 18+ messages in thread From: Lawrence Velázquez @ 2022-11-19 19:50 UTC (permalink / raw) To: Ray Andrews; +Cc: zsh-users On Sat, Nov 19, 2022, at 2:12 PM, Ray Andrews wrote: > Excellent. Yes, that's understandable. Main thing is to realize > that 'tree' want's two arguments there, not the intuitive single argument. A "single argument" is not intuitive at all. You would not run tree '-L 2' or tree -L\ 2 so it does not make any sense to think that "tree" wants a single argument there. -- vq ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-19 19:50 ` Lawrence Velázquez @ 2022-11-19 22:21 ` Ray Andrews 2022-11-20 8:55 ` Stephane Chazelas 0 siblings, 1 reply; 18+ messages in thread From: Ray Andrews @ 2022-11-19 22:21 UTC (permalink / raw) To: zsh-users On 2022-11-19 11:50, Lawrence Velázquez wrote: > > A "single argument" is not intuitive at all. You would not run > > tree '-L 2' > > or > > tree -L\ 2 > > so it does not make any sense to think that "tree" wants a single > argument there. Intuition is subjective. Besides, usually such switches don't demand a space IIRC and it really is actually one argument, this is an exception. Even if the space is demanded, one might still think of a switch as a single semantic instruction even if it must be syntactically two words. A filename with a space in it is NOT broken in half, it's still one entity, so I'm thinking the same way. I myself was naively thinking of it as nothing more than a string of four characters to be dropped into another string of characters -- as simple as that. As if it was the command line. But commands do have their need to group characters into arguments so the invisible rules must be followed. ' -L 2 ' must be invisibly broken in half. Which, interestingly 'eval' seems to do automatically. It's a huge thing coming to understand that what to me might look like 'just' a string of characters, to a command, needs to be viewed differently. Once I understand that, things get much simpler. I guess internally zsh must have all sorts of meta data attached to arrays to keep track of what's joined/split to/from what else. It would be cool to be able to somehow see this invisible stuff, some way to see the invisible structure of a command line. But I'll be ready the next time something like this crops up. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-19 22:21 ` Ray Andrews @ 2022-11-20 8:55 ` Stephane Chazelas 2022-11-20 13:47 ` Ray Andrews 0 siblings, 1 reply; 18+ messages in thread From: Stephane Chazelas @ 2022-11-20 8:55 UTC (permalink / raw) To: Ray Andrews; +Cc: zsh-users 2022-11-19 14:21:32 -0800, Ray Andrews: [...] > Intuition is subjective. Besides, usually such switches don't demand a > space IIRC and it really is actually one argument, this is an exception. > Even if the space is demanded, one might still think of a switch as a single > semantic instruction even if it must be syntactically two words. A filename > with a space in it is NOT broken in half, it's still one entity, so I'm > thinking the same way. I myself was naively thinking of it as nothing more > than a string of four characters to be dropped into another string of > characters -- as simple as that. As if it was the command line. But > commands do have their need to group characters into arguments so the > invisible rules must be followed. ' -L 2 ' must be invisibly broken in > half. Which, interestingly 'eval' seems to do automatically. It's a huge > thing coming to understand that what to me might look like 'just' a string > of characters, to a command, needs to be viewed differently. Once I > understand that, things get much simpler. I guess internally zsh must have > all sorts of meta data attached to arrays to keep track of what's > joined/split to/from what else. It would be cool to be able to somehow see > this invisible stuff, some way to see the invisible structure of a command > line. But I'll be ready the next time something like this crops up. [...] Just to clarify, when you enter: tree -L 1 or: eval 'tree -L 1' /usr/bin/tree ends up being executed but it doesn't receive any space character in any of the 3 arguments it receives. Not anymore that it receives coma characters when you enter the equivalent system "tree","-L","1"; perl code. Those spaces are part of the shell language syntax, just like the comas are part of the perl language syntax. It's the caller's role to pass tree, -L and 1 as separate arguments. What tree sees is argv[0] = "tree", argv[1] = "-L", argv[2] = "1". tree parses its options. It looks like it does not use the standard getopt() API or the GNU getopt_long() API for that: $ nm -D =tree | grep -i getopt $ ltrace -e '*opt*@*' tree > /dev/null +++ exited (status 0) +++ So it must be doing it by hand. If it were using getopt(), upon encountering "-L" in argv[1], it would look for the value for that -L option after the L in argv[1], and since here there's nothing, it would look for it in argv[2], if there was no argv[2], getopt() would fail and an error would be reported. tree is not using getopt() and it looks like it only accepts values for options that take values in the next argument. But even if it used getopt(), in: tree '-L 1' tree $'-L\x201' tree -L\ 1 arg='-L 1'; tree $arg (or system("tree", "-L 1") in perl) "tree" would be able to find a value for that -L option but that value would be " 1", not "1", same as if you had called it with -L ' 1' (or '--level= 1' or --level=' 1' or --level ' 1' if it were using GNU getopt_long()) and depending on how it decodes strings into numbers, it could very well choke on that leading space. As it happens, here it uses strtoul() to do that and doesn't even seem to check that the input string is fully decoded, so leading whitespace are ignored and it's very liberal as to what it accepts: $ ltrace -e '*strto*[lif]@*' tree -L $'\x20\t\r\n123.5e+20 whatever' > /dev/null tree->strtoul(" \t\r\n123.5e+20 whatever", 0, 0) = 123 +++ exited (status 0) +++ (it got 123 out of that <space><tab><cr><lf>123.5e+20<space>whatever argument. In any case zsh (or perl) has no saying on how tree may parse its options. You have to invoke it the way its meant to be invoked and in the case of tree, the option arguments must be passed as separate arguments. For most other commands, you would be able to pass them either as one -L1 argument or two -L and 1 arguments and also be able to combine more than one option in a single argument as in -xyL1 (assuming -x and -y are options that don't take arguments, are flags/boolean). -- Stephane ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-20 8:55 ` Stephane Chazelas @ 2022-11-20 13:47 ` Ray Andrews 2022-11-20 15:08 ` Stephane Chazelas 0 siblings, 1 reply; 18+ messages in thread From: Ray Andrews @ 2022-11-20 13:47 UTC (permalink / raw) To: zsh-users On 2022-11-20 00:55, Stephane Chazelas wrote: It's the caller's role to pass tree, -L and 1 as separate > arguments. Yeah I get it. The first time I 'got it' was when discussing 'aptitude' and the various quotings needed. Again one is creating an invisible level of organization imposed on a string of characters. Where it got confusing was when zsh imposed it's own ideas of grouping, making '-L 2' into a single entity. Dunno if it's really a thing to be desired but naively one might like some way of assembling a command string as if it was at CLI, that is *just* a string of characters -- which is how I was looking at it. > tree parses its options. It looks like it does not use the > standard getopt() API or the GNU getopt_long() API for that: > > $ nm -D =tree | grep -i getopt > $ ltrace -e '*opt*@*' tree > /dev/null > +++ exited (status 0) +++ What are you doing there? I have no 'nm' command here. No such command in Debian repository. > > So it must be doing it by hand. Yeah, so much in the GNU/Linux world is ad hoc. Everybody did their own thing. No rules. > In any case zsh (or perl) has no saying on how tree may parse > its options. You have to invoke it the way its meant to be > invoked and in the case of tree, the option arguments must be > passed as separate arguments. Sure. The main thing is to just understand that. Not too difficult when you do, something might need splitting or joining but it won't take long to figure out. But why does calling 'eval' fix it? It seems as if eval 'flattens' everything -- one is back to a command line string of characters with no imposed grouping. > > For most other commands, you would be able to pass them either > as one -L1 argument or two -L and 1 arguments and also be able > to combine more than one option in a single argument as in > -xyL1 (assuming -x and -y are options that don't take > arguments, are flags/boolean). > Yeah, as I was saying, it does seem that 'tree' is very crabby. Why don't the GNU people iron these things out? ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-20 13:47 ` Ray Andrews @ 2022-11-20 15:08 ` Stephane Chazelas 2022-11-20 16:27 ` Ray Andrews 0 siblings, 1 reply; 18+ messages in thread From: Stephane Chazelas @ 2022-11-20 15:08 UTC (permalink / raw) To: Ray Andrews; +Cc: zsh-users 2022-11-20 05:47:33 -0800, Ray Andrews: [...] > > arguments. > Yeah I get it. The first time I 'got it' was when discussing 'aptitude' and > the various quotings needed. Again one is creating an invisible level of > organization imposed on a string of characters. Where it got confusing was > when zsh imposed it's own ideas of grouping, making '-L 2' into a single > entity. zsh doesn't impose anything. "-L 2" is a string made of 4 characters in any shell or programming language. There's no programming language where "-L 2" means 2 strings -L and 2. What may be the source of your confusion is that in the Bourne shell, there was a second round of splitting applied to unquoted text in list context. Space is a special character in the syntax of the shell which is used to delimit command arguments, but it's also in the default value of the $IFS special variable which was used to *further* split text into futher arguments. In the Bourne shell (introduced in the late 70s): edit file Is like in all other shells code that's meant to run /path/to/edit with "edit" and "file" as separate arguments, but if $IFS also happens to contain "i", then those two words happen to be further split into "ed", "t" and "f", "le", so you end up running "/bin/ed" with "ed", "t", "f" and "le" as arguments. That splitting also happened on top of parameter expansion. ed $file Upon syntax evaluation yields "ed" and "$file" as two separate tokens, but both "ed" and the contents of $file further underwent $IFS-splitting. Almost worse, globbing also happened on top of $file expansion. That is a very weird feature from a language design point of view. Anybody who doesn't know what the shells looked like before the Bourne shell would think Stephen Bourne was out of his mind. The thing is in the original Unix shell (designed in the early 70s on computers than had kilobytes of RAM), there was no variable. There was a concept of script that could take arguments, and in the script, you'd refer to them as $1, $2... the positional parameters (which have survived in modern shells). Parameter expansion was really crude, it was a bit like aliases, the contents of the parameter was just expanded in place into the code being evaluated. So for instance, if you called your script as: my-script 'hi;rm -rf /' And the script did: echo $1 That would say hi and destroy the system. It had its uses though. my-script 'dir1 dir2' 'file3 file4' With a script that did: ls $1 rm $2 Would list the dir1 and dir2 directories and remove the file3 and file4 files. And: my-script '*.txt' in a script that did: cd /foo rm $1 cd /bar rm $1 Would remove the files with name ending in .txt in both /foo and /bar (or the current directory if cd failed... but that's beside the point). My understanding is that the Bourne shell's bizarre IFS handling and the fact that globbing was performed upon parameter expansion was an attempt to keep some level of backward compatibility with that Thompson shell. You see similar things hapenning in csh from the same era. The Korn shell (from the early 80s) kept most of that with the exception that it only did IFS-splitting upon expansions, not on literal text. In: file=/some/file edit $file With IFS=i, $file would be split into /some/f and le, but edit would stay edit. The POSIX specification of sh standardised that behaviour so that's the one found in bash / dash / yash... as well. That nonsense was fixed in most shells written after that though, starting with rc, the shell of plan9 / Unix v10 from the late 80s, but also zsh (early 90s) or more recently (mid 2000s) fish. The fact that in sh/ksh/bash: arg='-L 1' tree $arg Calls tree with "-L" and "1" as arguments is not that bash doesn't do some sort of magic "grouping" that zsh would be doing, but that ksh/bash contrary to zsh does that extra layer of $IFS-splitting from the Bourne shell on top of the syntax parsing as the default value of $IFS happens to contain the space character. That's why in sh/bash/ksh you almost always need to quote parameter expansions if you intend to pass the contents of a variable as an argument to a command. See https://unix.stackexchange.com/questions/171346/security-implications-of-forgetting-to-quote-a-variable-in-bash-posix-shells for the kind of thing that can happen if you forget. > Dunno if it's really a thing to be desired but naively one might > like some way of assembling a command string as if it was at CLI, that is > *just* a string of characters -- which is how I was looking at it. If you want to assemble a string, and that string to be evaluated as shell code, that's what eval is for. eval evaluate code written in the shell language. That's a way to dynamically invoke the language interpreter. But generally, that's not what you want. Writing correct shell code dynamically based on external input is very easy to get wrong. In your case, it rather looks like you want to build up a list of arguments to pass to a command for which you obviously need a shell list/array variable. > > tree parses its options. It looks like it does not use the > > standard getopt() API or the GNU getopt_long() API for that: > > > > $ nm -D =tree | grep -i getopt > > $ ltrace -e '*opt*@*' tree > /dev/null > > +++ exited (status 0) +++ > What are you doing there? I have no 'nm' command here. No such command in > Debian repository. nm is a standard development command (though not -D which AFAIK is a GNU extension). Part of GNU binutils on GNU systems. Here used to list the external functions that the utility claims it needs. The above shows that tree doesn't use the getopt() standard command or that if it does, it embedded a copy into the executable rather than using the one from the GNU libc. > > So it must be doing it by hand. > Yeah, so much in the GNU/Linux world is ad hoc. Everybody did their own > thing. No rules. Note that tree is not part of the GNU project, it's just a utility written by some guy and shared to the world. There is *some* level of consistency among utilities in the GNU toolchest. There is even such a thing as published GNU coding standards. See for instance https://www.gnu.org/prep/standards/html_node/Command_002dLine-Interfaces.html#Command_002dLine-Interfaces zsh is not part of the GNU project either. Note that zsh also predates Linux and has been used in a wide variety of GNU and non-GNU systems. I'll agree with you that the lack of consistency in API between the different tools that are available out there can be annoying (nothing to do with GNU or Linux) [...] > figure out. But why does calling 'eval' fix it? It seems as if eval > 'flattens' everything -- one is back to a command line string of characters > with no imposed grouping. eval evaluates shell code. I struggle to understand what you don't understand. Maybe you're thinking too much or too little of what a "command line string" is. A "command line string" like: var='value'; if blah; then echo $(cmd) | tr -d x; fi is just one line of code in the syntax of the shell programing language. It's not some magic universal language to talk to the system. You can store that code in bits in variables with: a="'val" b="ue'; if bl" c='ah; then ech' d='o $(cmd) | tr -' e='d x; fi' But surely you don't expact var=$a$b$c$d$e to have the same effect as running that command. You can however do: eval "var=$a$b$c$d$e" For that string to be passed as code to the shell language interpreter for it to interpret. [...] > Yeah, as I was saying, it does seem that 'tree' is very crabby. Why don't > the GNU people iron these things out? [...] Again, tree has nothing to do with the GNU project. There is also unfortunatly not one standard to parse options and arguments. There is POSIX getopt() but for instance, it doesn't support long options (neither a la GNU nor a la X11, or a la perl Getopt::Long nor a la zsh zparseopts...). -- Stephane ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-20 15:08 ` Stephane Chazelas @ 2022-11-20 16:27 ` Ray Andrews 2022-11-20 20:16 ` Stephane Chazelas 0 siblings, 1 reply; 18+ messages in thread From: Ray Andrews @ 2022-11-20 16:27 UTC (permalink / raw) To: zsh-users On 2022-11-20 07:08, Stephane Chazelas wrote: > zsh doesn't impose anything. "-L 2" is a string made of 4 > characters in any shell or programming language. There's no > programming language where "-L 2" means 2 strings -L and 2. But that's the thing, I'd naively thought the variable could be inserted in the command string and be 'just' a string of characters, but zsh imposes that 'tree' must see '-L 2' as a single entity, yes? As shown, we need word splitting to solve the problem and show tree what it wants to see. Or, as I was speculating, some way of just flattening the string back to nothing more than a string of characters -- but then again, probably tree wouldn't like that either, perhaps the string is always 'packaged' into words? If so, then there's no avoiding that we must word-split '-L 2' into two words and there's nothing to wish for. > That is a very weird feature from a language design point of > view. Anybody who doesn't know what the shells looked like > before the Bourne shell would think Stephen Bourne was out of > his mind. > It's what happens when functionality expands by accretion -- simple structures no longer suffice but they are tweaked rather than rethought. It's like the difference between the layout of Paris vs. London. > My understanding is that the Bourne shell's bizarre IFS handling > and the fact that globbing was performed upon parameter > expansion was an attempt to keep some level of backward > compatibility with that Thompson shell. You see similar things > hapenning in csh from the same era. The dilemma of compatibility! No easy answers. I myself tend towards a 'garage to the dump' mentality but it's not that simple. > The Korn shell (from the early 80s) kept most of that with the > exception that it only did IFS-splitting upon expansions, not on > literal text. Holy cow! IFS splitting in text?? > The fact that in sh/ksh/bash: > > arg='-L 1' > tree $arg > > Calls tree with "-L" and "1" as arguments is not that bash > doesn't do some sort of magic "grouping" that zsh would be > doing, but that ksh/bash contrary to zsh does that extra layer > of $IFS-splitting from the Bourne shell on top of the syntax > parsing as the default value of $IFS happens to contain the > space character. Ah! So my little issue is a zsh exclusive? Not complaining tho, I don't like zsh doing things I didn't ask it to do so if in this case I need to request a word split that's just fine. After all it's tree that's being difficult so the problem is rare and easily solved. > That's why in sh/bash/ksh you almost always need to quote > parameter expansions if you intend to pass the contents of a > variable as an argument to a command. > > See > https://unix.stackexchange.com/questions/171346/security-implications-of-forgetting-to-quote-a-variable-in-bash-posix-shells > for the kind of thing that can happen if you forget. Heavy. That's the closest I've come to understanding those vulnerabilities. > > If you want to assemble a string, and that string to be > evaluated as shell code, that's what eval is for. eval evaluate > code written in the shell language. That's a way to dynamically > invoke the language interpreter. Ok, so I understood that correctly. At least partially. > What are you doing there? I have no 'nm' command here. No such command in > Debian repository. > nm is a standard development command (though not -D which AFAIK > is a GNU extension). Part of GNU binutils on GNU systems. Ah, that's why I can't find it. It's hard to find commands so packaged. > Note that tree is not part of the GNU project, it's just a > utility written by some guy and shared to the world. There is > *some* level of consistency among utilities in the GNU > toolchest. There is even such a thing as published GNU coding standards. > See for instance > https://www.gnu.org/prep/standards/html_node/Command_002dLine-Interfaces.html#Command_002dLine-Interfaces Yeah, tides of organization wash here and there. No overall commander tho, so a bit of chaos is unavoidable. > I'll agree with you that the lack of consistency in API between the > different tools that are available out there can be annoying > (nothing to do with GNU or Linux) You know, half of it is just cultural adaptation. When I first started with Linux the first thing I did was decide between zsh and bash and I'm basically assuming that Linux will be like DOS but slightly more powerful and zsh will be like C and that all Linux OS apps will be as flawless as WordPerfect 5.1. CRASH! Now that I'm used to being in a bazaar not a monastery it's easier to get around. One does get mud on one's shoes. > eval evaluates shell code. I struggle to understand what you > don't understand. Maybe you're thinking too much or too little > of what a "command line string" is. A "command line string" > like: I think at this level of my education it's enough to say that eval looks at it's arguments as if they were typed on the command line. I haven't really 'understood' anything, I use eval ad hoc when it seems to solve problems. Using it with real understanding is a future goal. > For that string to be passed as code to the shell language > interpreter for it to interpret. Right, I get that: 2 /aWorking/Zsh/Source/Wk 0 $ var="echo howdy" 2 /aWorking/Zsh/Source/Wk 0 $ eval $var howdy > Again, tree has nothing to do with the GNU project. I dunno, I get it all from Debian and Debian is a GNU/Linux OS so I don't know more than that. Thanks for a most educational post! I love these history lessons, they inform my entire outlook. You can't understand where you are unless you understand where you've been. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-20 16:27 ` Ray Andrews @ 2022-11-20 20:16 ` Stephane Chazelas 2022-11-20 22:31 ` Bart Schaefer 2022-11-20 22:47 ` Ray Andrews 0 siblings, 2 replies; 18+ messages in thread From: Stephane Chazelas @ 2022-11-20 20:16 UTC (permalink / raw) To: Ray Andrews; +Cc: zsh-users 2022-11-20 08:27:01 -0800, Ray Andrews: > > On 2022-11-20 07:08, Stephane Chazelas wrote: > > zsh doesn't impose anything. "-L 2" is a string made of 4 > > characters in any shell or programming language. There's no > > programming language where "-L 2" means 2 strings -L and 2. > > But that's the thing, I'd naively thought the variable could be inserted in > the command string and be 'just' a string of characters, but zsh imposes > that 'tree' must see '-L 2' as a single entity, yes? No, you have it completely backwards. And I think at the heart of your misunderstanding there is the misconception that you pass a "command string" to an executable. You do not, you pass a list of arguments. On Unix-like systems, execution of commands are with the execve() system call which takes 3 arguments: 1. the path name to the executable 2. a list of arguments: argv[] 3. a list of environment variables: envp[] What is commonly refered to as a command line is the shell's interface to that system call. VAR1=foo VAR2=bar cmd 'arg 1' arg2 Is how you tell your shell to do a execve("/path/to/cmd", ["cmd", "arg 1", "arg2"], [exportedvariable, "VAR1=foo", "VAR2=bar"]) You do not pass the "cmd 'arg 1' arg2" string as argument to /path/to/cmd. > As shown, we need word > splitting to solve the problem and show tree what it wants to see. Or, as I > was speculating, some way of just flattening the string back to nothing more > than a string of characters -- but then again, probably tree wouldn't like > that either, perhaps the string is always 'packaged' into words? If so, > then there's no avoiding that we must word-split '-L 2' into two words and > there's nothing to wish for. Why store those two arguments as a string with a space in between them and then ask the shell to split it, that makes no sense. Just store those two arguments and pass them to tree, that's exactly what arrays are for. In several shells like rc or fish, all variables are arrays (also in csh, but csh has many other issues of its own). From a design point of view, that's the most sensible thing to do as what shells deal with primarily is the list of arguments to commands¹ zsh: args=(-L 1); tree $args rc: args=(-L 1); tree $args fish: set args -L 1; tree $args tcsh: set args = (-L 1); tree $args:q ksh: set -A args -- -L 1; tree "${args[@]}" ksh93/bash2+(/zsh): args=(-L 1); tree "${args[@]}" (¹ a complication and a spanner in the works is that environment variables (those VAR=value strings passed in envp[] above) are string/scalar only; in csh and in the Bourne shell, shell variables and env variables were more separated than they are in modern shells). If you wish arg='-L 2' tree $arg passed "-L" and "2" as separate arguments to tree, do you also wish that: file='my file.txt' rm -- $file Also passed "my" and "file.txt" as separate arguments to rm for rm to unlink those 2 files? What about: files=( 'my file.txt' 'my other file.txt' ) rm -- $files Should they be split too? How about: text='shutdown|reboot' echo $text Should that run echo shutdown|reboot and reboot? [...] > > Calls tree with "-L" and "1" as arguments is not that bash > > doesn't do some sort of magic "grouping" that zsh would be > > doing, but that ksh/bash contrary to zsh does that extra layer > > of $IFS-splitting from the Bourne shell on top of the syntax > > parsing as the default value of $IFS happens to contain the > > space character. > Ah! So my little issue is a zsh exclusive? No, again, any recent shell (i.e. post 1980) that had no intention of being compatible with the Bourne shell including rc (and derivatives like akanga or es) and fish behave like zsh in this instance². In fish, set arg '-L 1' tree $arg Or in rc/es/akanga: arg = '-L 1' tree $arg all call tree with one "-L 1" argument. (² and even better, zsh still has that issue that unquoted expansions to empty removal, one of the issues with Bourne-like shells that it hasn't fixed. rc/es/akanga/fish have). > Not complaining tho, I don't > like zsh doing things I didn't ask it to do so if in this case I need to > request a word split that's just fine. After all it's tree that's being > difficult so the problem is rare and easily solved. No, it's not tree being difficult. In any command doing cmd '-o arg' is passing the " arg" string as argument to the -o option. sort '-o file' file Will sort the "file" file into the " file" file just like you asked. If you want to sort it into itself, it's sort -o file file [...] > > nm is a standard development command (though not -D which AFAIK > > is a GNU extension). Part of GNU binutils on GNU systems. > Ah, that's why I can't find it. It's hard to find commands so packaged. apt-file is your friend. For the POSIX standard (2018 edition) see https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/ [...] > > Again, tree has nothing to do with the GNU project. > > I dunno, I get it all from Debian and Debian is a GNU/Linux OS so I don't > know more than that. On Debian dpkg -S =tree Will tell you what package the tree command comes from. apt showsrc that-package Will tell you how it was built and generally where the source came from. Here, not from gnu.org but from http://mama.indstate.edu/users/ice/tree/ http://mama.indstate.edu/users/ice/tree/changes.html shows it has a long history and might be even as old as zsh. -- Stephane ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-20 20:16 ` Stephane Chazelas @ 2022-11-20 22:31 ` Bart Schaefer 2022-11-21 2:13 ` Ray Andrews 2022-11-20 22:47 ` Ray Andrews 1 sibling, 1 reply; 18+ messages in thread From: Bart Schaefer @ 2022-11-20 22:31 UTC (permalink / raw) To: Ray Andrews, zsh-users On Sun, Nov 20, 2022 at 12:17 PM Stephane Chazelas <stephane@chazelas.org> wrote: > > No, you have it completely backwards. And I think at the heart > of your misunderstanding there is the misconception that you > pass a "command string" to an executable. This misconception may be an effect of Ray's background with DOS/Windows command interpreters. In that world, each command is responsible both for parsing its "arguments" and for interpreting them. There was (is, I suppose) a standard DLL to support the parsing part, and a lot of commands used it, but it was still down to the command itself to call that DLL. In the Unix-based world, it's expected that the shell will do the parsing and the command only has to do the interpreting, so there are a lot more rules about the structure of the command line. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-20 22:31 ` Bart Schaefer @ 2022-11-21 2:13 ` Ray Andrews 0 siblings, 0 replies; 18+ messages in thread From: Ray Andrews @ 2022-11-21 2:13 UTC (permalink / raw) To: zsh-users > In the Unix-based world, it's expected that the shell will do the > parsing and the command only has to do the interpreting, so there are > a lot more rules about the structure of the command line. Hit the nail on the head :-) ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: why is eval needed? 2022-11-20 20:16 ` Stephane Chazelas 2022-11-20 22:31 ` Bart Schaefer @ 2022-11-20 22:47 ` Ray Andrews 1 sibling, 0 replies; 18+ messages in thread From: Ray Andrews @ 2022-11-20 22:47 UTC (permalink / raw) To: zsh-users On 2022-11-20 12:16, Stephane Chazelas wrote: > >> But that's the thing, I'd naively thought the variable could be inserted in >> the command string and be 'just' a string of characters, but zsh imposes >> that 'tree' must see '-L 2' as a single entity, yes? > No, you have it completely backwards. But everything so far seems to tell me that I'm right. When '-L 2' is presented as two arguments instead of one, everything is fine. Are we miscommunicating? > And I think at the heart > of your misunderstanding there is the misconception that you > pass a "command string" to an executable. > > You do not, you pass a list of arguments. > I know, but at least on the command line one can think of it as a string. I understand that it it will be parsed as 'words' (including spaces if quoted). > > Why store those two arguments as a string with a space in between them > and then ask the shell to split it, that makes no sense. I know that now. But as we discussed, the normal thing is for a switch to present as a single argument so that's what I did. I quite understand the mistake NOW. > (¹ a complication and a spanner in the works is that environment > variables (those VAR=value strings passed in envp[] above) are > string/scalar only; in csh and in the Bourne shell, shell variables and > env variables were more separated than they are in modern shells). Awesome that you can keep all this stuff in memory and not get your wires crossed. > > What about: > > files=( > 'my file.txt' > 'my other file.txt' > ) > rm -- $files > > Should they be split too? No, I understand that well. I think this is a closed issue, I think I'm quite content. > Ah! So my little issue is a zsh exclusive? > No, again, any recent shell Ok, not 'exclusive' but it's been a point of difference between at least zsh and bash. > No, it's not tree being difficult. In any command doing Bbbbut we've agreed that tree is being difficult! You've explained it in great detail. > apt-file is your friend. So it is! > Will tell you what package the tree command comes from. > > apt showsrc that-package It's just not something I need to worry about. To me it's all 'Linux'. Maybe one day I'll download some sources (if in C) and start tinkering. For now it just doesn't matter. The thing is to get a bit more competent with zsh. ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2022-11-21 2:16 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-11-19 14:34 why is eval needed? Ray Andrews 2022-11-19 14:43 ` Roman Perepelitsa 2022-11-19 17:02 ` Ray Andrews 2022-11-19 17:10 ` Roman Perepelitsa 2022-11-19 18:02 ` Clinton Bunch 2022-11-19 18:18 ` Roman Perepelitsa 2022-11-19 16:48 ` Stephane Chazelas 2022-11-19 19:12 ` Ray Andrews 2022-11-19 19:50 ` Lawrence Velázquez 2022-11-19 22:21 ` Ray Andrews 2022-11-20 8:55 ` Stephane Chazelas 2022-11-20 13:47 ` Ray Andrews 2022-11-20 15:08 ` Stephane Chazelas 2022-11-20 16:27 ` Ray Andrews 2022-11-20 20:16 ` Stephane Chazelas 2022-11-20 22:31 ` Bart Schaefer 2022-11-21 2:13 ` Ray Andrews 2022-11-20 22:47 ` Ray Andrews
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).