for loop question

for loop question

[Ray Andrews]

test()
{

     # THIS WORKS  FINE:

     i=1
     while [ -n "$TLC[i]" ]; do
       let i=i+1
       print -ru2 $TLC[i]
     done

     # THIS WORKS FINE:

     for ((i=1; i<6; i++))
     do
       print -ru2 $TLC[i]
     done

     # ... BUT THIS FAILS WITH "BAD OUTPUT FORMAT SPECIFICATION":

     for ((i=1; [ -n "$TLC[i]" ]; i++))
     do
       print -ru2 $TLC[i]
     done
}

I've fiddled around with various modifications but it seems to dislike " 
[ -n "$TLC[i]" ] "
even though the 'while' loop is happy with it.  That expression 
evaluates to true or
false, does it not? So why won't 'for' swallow it?

Re: for loop question

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 396 bytes --]

> I've fiddled around with various modifications but it seems to dislike "
[ -n "$TLC[i]" ] "
> even though the 'while' loop is happy with it.  That expression evaluates
to true or
> false, does it not? So why won't 'for' swallow it?

The double parents (( )) are special arithmetic syntax.  You can't use an
ordinary shell command like "test" ( for which "[" is an alias) inside that
construct.

Re: for loop question

[Stephane Chazelas]

2014-11-02 13:00:14 -0800, Bart Schaefer:
> > I've fiddled around with various modifications but it seems to dislike "
> [ -n "$TLC[i]" ] "
> > even though the 'while' loop is happy with it.  That expression evaluates
> to true or
> > false, does it not? So why won't 'for' swallow it?
> 
> The double parents (( )) are special arithmetic syntax.  You can't use an
> ordinary shell command like "test" ( for which "[" is an alias) inside that
> construct.

You could with:

for ((i=1; (z[$([ -n "$TLC[i]" ])0]),$? == 0; i++))
  print -ru2 -- $TLC[i]

(not that you would want to).

Here, you more likely want:

for i ("$TLC[@]") print -ru2 -- $i

or

print -rlu2 -- "$TLC[@]"

or:

for ((i = 1; i <= $#TLC; i++)) print -ru2 -- $TLC[i]


Or (to print only till the first empty element):

for ((i = 1; $#TLC[i]; i++)) print -ru2 -- "$TLC[i]"

Or:

print -rlu2 -- "${(@)TLC[1,TLC[(i)]-1]}"

-- 
Stephane

Re: for loop question

[Ray Andrews]

On 11/02/2014 01:37 PM, Stephane Chazelas wrote:
> You could with:
>
> for ((i=1; (z[$([ -n "$TLC[i]" ])0]),$? == 0; i++))
>    print -ru2 -- $TLC[i]
>
> (not that you would want to).
No.  That is pure sadism ;-) ;-)

But it does show that 'for ((' CAN stop and digest  ' [ -n "$TLC[i]" ] 
'  if it wants too,
it just has to make it obscenely difficult. Why can't the truth test of 
a command just be taken
as 'arithmetic' plain and simple?

      $ for ((; 1 ;)) echo true!
      true!
      true!
      true!
      .....

      $ for ((; 0 ;)) echo true!
[nothing]

.... so why/how is it that the return value of  a ' [] ' test is NOT 
either 1 or 0?


Thanks for these, there is much  to learn from them:
> Here, you more likely want:
>
> for i ("$TLC[@]") print -ru2 -- $i
>
> or
>
> print -rlu2 -- "$TLC[@]"
>
> or:
>
> for ((i = 1; i <= $#TLC; i++)) print -ru2 -- $TLC[i]
>
>
> Or (to print only till the first empty element):
>
> for ((i = 1; $#TLC[i]; i++)) print -ru2 -- "$TLC[i]"
>
> Or:
>
> print -rlu2 -- "${(@)TLC[1,TLC[(i)]-1]}"
>

Re: for loop question

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

On Nov 2, 2014 2:42 PM, "Ray Andrews" <rayandrews@eastlink.ca> wrote:
>
> But it does show that 'for ((' CAN stop and digest  ' [ -n "$TLC[i]" ] '
if it wants too,
> it just has to make it obscenely difficult. Why can't the truth test of a
command just be taken
> as 'arithmetic' plain and simple?

It's a matter of order-of-evaluation and contextual tokenization.  Once the
parser encounters "for ((" it stops looking for ordinary shell command
tokens and starts looking for math tokens; a word like "test" in that
context is a numeric variable name, not a command name, and "[" has another
meaning as well.  But "$" introduces an expansion in either context, so
$(...) switches back into the regular parsing mode.

Re: for loop question

[Oliver Kiddle]

Ray Andrews wrote:
> > You could with:
> >
> > for ((i=1; (z[$([ -n "$TLC[i]" ])0]),$? == 0; i++))
> >    print -ru2 -- $TLC[i]
> >
> > (not that you would want to).
> No.  That is pure sadism ;-) ;-)
> 
> But it does show that 'for ((' CAN stop and digest  ' [ -n "$TLC[i]" ] 
> '  if it wants too,
> it just has to make it obscenely difficult. Why can't the truth test of 
> a command just be taken
> as 'arithmetic' plain and simple?

Inside of (( ... )) is arithmetic context. So different syntax is valid
there. [ is just a command. Look at the error message you get if you put
it there:
  % for ((; [ -f foo ] ; )) echo true
  zsh: bad output format specification

Zsh sees the square brackets and in arithmetic context, square brackets
are used for selecting an output format, specifically a number base. For
example:
  % echo $(( [#2] 37 ))
  2#100101

There's also a command named true which just returns true. So what do
you think the following might do:

  % for ((; true ;)) echo true!

This actually looks for the variable $true and interprets it as a
number.

As you can see, the syntaxes conflict. While we could add a simple way
to escape back into normal syntax from arithmetic syntax, it'd be no
nicer than the existing alternatives.

Oliver

Re: for loop question

[Ray Andrews]

On 11/02/2014 03:24 PM, Oliver Kiddle wrote:
> As you can see, the syntaxes conflict. While we could add a simple way 
> to escape back into normal syntax from arithmetic syntax, it'd be no 
> nicer than the existing alternatives. 

> Oliver 
This is going to take so long to get comfortable with .  Anyway, you 
can't break the grammar.
It might look like it could be simple, but if it's a violation, then 
it's a violation.  For now I'll take it
on faith that this:

     (z[$([ -n "$TLC[i]" ])0]),$? == 0;

is as simple as it can be.  Practically speaking, tho, those other 
examples are very nice, so this is
just theoretical whining.

Re: for loop question

[Mikael Magnusson]

On Sun, Nov 2, 2014 at 10:37 PM, Stephane Chazelas
<stephane.chazelas@gmail.com> wrote:
> 2014-11-02 13:00:14 -0800, Bart Schaefer:
>> > I've fiddled around with various modifications but it seems to dislike "
>> [ -n "$TLC[i]" ] "
>> > even though the 'while' loop is happy with it.  That expression evaluates
>> to true or
>> > false, does it not? So why won't 'for' swallow it?
>>
>> The double parents (( )) are special arithmetic syntax.  You can't use an
>> ordinary shell command like "test" ( for which "[" is an alias) inside that
>> construct.
>
> You could with:
>
> for ((i=1; (z[$([ -n "$TLC[i]" ])0]),$? == 0; i++))
>   print -ru2 -- $TLC[i]
>
> (not that you would want to).
>
> Here, you more likely want:
>
> for i ("$TLC[@]") print -ru2 -- $i
>
> or
>
> print -rlu2 -- "$TLC[@]"
>
> or:
>
> for ((i = 1; i <= $#TLC; i++)) print -ru2 -- $TLC[i]
>
>
> Or (to print only till the first empty element):
>
> for ((i = 1; $#TLC[i]; i++)) print -ru2 -- "$TLC[i]"
>
> Or:
>
> print -rlu2 -- "${(@)TLC[1,TLC[(i)]-1]}"

We might want to avoid using obscure mixes of the discouraged
alternate syntax with syntax that depends on short_loops being set,
when helping people who are asking questions about basic syntax. :)

-- 
Mikael Magnusson

Re: for loop question

[Ray Andrews]

On 11/02/2014 05:53 PM, Mikael Magnusson wrote:
> We might want to avoid using obscure mixes of the discouraged 
> alternate syntax with syntax that depends on short_loops being set, 
> when helping people who are asking questions about basic syntax. :) 
It does get a bit overwhelming.  However, when I poke at the syntax by 
asking 'why can't we ...' sorts of questions, I expect to more or less 
have my fuses blown ;-)  Still, any sort of heads up about what is 
considered proper vs. what is considered discouraged will be most 
appropriate.

Re: for loop question

[Mikael Magnusson]

On Mon, Nov 3, 2014 at 3:22 AM, Ray Andrews <rayandrews@eastlink.ca> wrote:
> On 11/02/2014 05:53 PM, Mikael Magnusson wrote:
>>
>> We might want to avoid using obscure mixes of the discouraged alternate
>> syntax with syntax that depends on short_loops being set, when helping
>> people who are asking questions about basic syntax. :)
>
> It does get a bit overwhelming.  However, when I poke at the syntax by
> asking 'why can't we ...' sorts of questions, I expect to more or less have
> my fuses blown ;-)  Still, any sort of heads up about what is considered
> proper vs. what is considered discouraged will be most appropriate.

If you look up the section "Complex Commands" in the manpage, they're
all listed fairly well explained. The section after that is called
"Alternate Forms for Complex Commands" and lists some syntax that is
convenient interactively but not super encouraged. Using any syntax
that depends on SHORT_LOOPS being set will of course not work in any
other shell, and also has the downside that making a small mistake
will usually still parse but do something entirely unexpected. I'm a
bit more opposed to SHORT_LOOPS than others here, but when I disabled
the option, I immediately found like 5 bugs in the completion system
so I think I'm at least somewhat justified :). The whole point of it
is just to not have to write do; done, but it causes so many bugs and
confusion.

As to the for loop, it has two different forms. They're almost so
different you could say the only thing they have in common is the word
"for" :). One is
for some variables in list of values; do
  list; of; commands using $some $variables
done
Think of that as a foreach loop. The other form is
for (( i=0; i<10; i++ )); do
  list; of; commands using $i
done
This form is exclusively for math expressions, you can't just stick
arbitrary commands in there. If you want that, just use a while loop.
commands to initialize state
while my condition; do
  stuff
  here
  increment iterator or whatever
done
because as you saw, stuffing it all into the for line would just get
really ugly. And you want the foreach form for your initial example as
Stephane said, there is no need to iterate over the index unless you
have a veeery large array, and then it will already be very slow
anyway.

As a side note, if you ever want a do-while loop instead of a regular
while loop, the trick is
while just put the whole body;
      of the loop here instead;
      and the last command is;
      the terminating condition; do done

-- 
Mikael Magnusson

Re: for loop question

[Ray Andrews]

On 11/03/2014 01:22 AM, Mikael Magnusson wrote:
> If you look up the section "Complex Commands" in the manpage, they're
> all listed fairly well explained. ...

Thanks.  You know, with so many documents available, it's hard to know 
what it worth
reading.  I wish there was a sort of recommended list of docs and web 
sites so that one
wouldn't waste so much time looking for things worth reading. I have 
Oliver's book
here, and at least I know it's approved.
> As to the for loop, it has two different forms. They're almost so
> different you could say the only thing they have in common is the word
> "for" :). One is
Just those three lines might save a fella hours of grief.  I intuitively 
expect them to be
related *and* to be related to the C form, which explains my head 
banging over:

     for ((i = 1; [ -n ${TLC[i] ]; i++))

... because in C one can always test for the return value of anything at 
all and it looks so
very much like C.

     for ((i=1; (z[$([ -n "$TLC[i]" ])0]),$? == 0; i++))

... is sick and twisted, but:

     for ((i = 1; $#TLC[i]; i++))

... is sweetly, intuitively best anyway.
> As a side note, if you ever want a do-while loop instead of a regular
> while loop, the trick is
> while just put the whole body;
>        of the loop here instead;
>        and the last command is;
>        the terminating condition; do done
Marvelous, I was wondering about that.

If I ever master zsh I'd like to write an essay: "Zsh culture for the 
sweet and innocent: Truths you hold to be self-evident that aren't true 
any more, or the weltanschauung of weltschmertz"

Re: for loop question

[Han Pingtian]

On Sun, Nov 02, 2014 at 09:37:13PM +0000, Stephane Chazelas wrote:
> 2014-11-02 13:00:14 -0800, Bart Schaefer:
> > > I've fiddled around with various modifications but it seems to dislike "
> > [ -n "$TLC[i]" ] "
> > > even though the 'while' loop is happy with it.  That expression evaluates
> > to true or
> > > false, does it not? So why won't 'for' swallow it?
> > 
> > The double parents (( )) are special arithmetic syntax.  You can't use an
> > ordinary shell command like "test" ( for which "[" is an alias) inside that
> > construct.
> 
> You could with:
> 
> for ((i=1; (z[$([ -n "$TLC[i]" ])0]),$? == 0; i++))
>   print -ru2 -- $TLC[i]
> 
What does the (z[$([ -n "$TLC[i]" ])0]) mean, please?

Re: for loop question

[Han Pingtian]

On Tue, Nov 04, 2014 at 09:56:39AM +0800, Han Pingtian wrote:
> On Sun, Nov 02, 2014 at 09:37:13PM +0000, Stephane Chazelas wrote:
> > 2014-11-02 13:00:14 -0800, Bart Schaefer:
> > > > I've fiddled around with various modifications but it seems to dislike "
> > > [ -n "$TLC[i]" ] "
> > > > even though the 'while' loop is happy with it.  That expression evaluates
> > > to true or
> > > > false, does it not? So why won't 'for' swallow it?
> > > 
> > > The double parents (( )) are special arithmetic syntax.  You can't use an
> > > ordinary shell command like "test" ( for which "[" is an alias) inside that
> > > construct.
> > 
> > You could with:
> > 
> > for ((i=1; (z[$([ -n "$TLC[i]" ])0]),$? == 0; i++))
> >   print -ru2 -- $TLC[i]
> > 
> What does the (z[$([ -n "$TLC[i]" ])0]) mean, please?
> 
I think I have got the meaning of it. The goal is to run 
'[ -n "$TLC[i]" ]' in a math context, so embed the testing in the
subscript of 'z[..]'. The 'z' will be explained as a array in this math
context, the value of '$z[...]' is ignored, only the result of 
'[ -n "$TLC[i]" ]' is compared with 0.

Re: for loop question

[Bart Schaefer]

On Nov 4,  9:56am, Han Pingtian wrote:
}
} > for ((i=1; (z[$([ -n "$TLC[i]" ])0]),$? == 0; i++))
} >   print -ru2 -- $TLC[i]
} > 
} What does the (z[$([ -n "$TLC[i]" ])0]) mean, please?

It's an overly-obfuscated way to write $([ -n "$TLC[i]" ]).  The real
work is the ",$?" part.

z[...],$? is used to throw away the z[...] part and keep the value of $?.

The value of $? comes from the side-effect of the $(...) inside the
subscript of z[...].

The 0 is appended because [ -n "$TLC[i]" ] does not produce any output,
but there has to be a valid number as the subscript, hence z[0].

This could be written a LOT more plainly as

    for ((i=1; $([ -n "$TLC[i]" ]; echo $?) == 0; i++))

without needing any more characters.  If we're playing golf,

    for ((i=1; ! $([ "$TLC[i]" ]; echo $?); i++))

works too.

Which, by the way, points out another reason you can't just use the
exit status of a command in a math context:  command success/failure
are the reverse of true/false when taken as integer values.

Re: for loop question

[Han Pingtian]

On Mon, Nov 03, 2014 at 06:43:38PM -0800, Bart Schaefer wrote:
> } What does the (z[$([ -n "$TLC[i]" ])0]) mean, please?
> 
> It's an overly-obfuscated way to write $([ -n "$TLC[i]" ]).  The real
> work is the ",$?" part.
> 
> z[...],$? is used to throw away the z[...] part and keep the value of $?.
> 
> The value of $? comes from the side-effect of the $(...) inside the
> subscript of z[...].
> 
> The 0 is appended because [ -n "$TLC[i]" ] does not produce any output,
> but there has to be a valid number as the subscript, hence z[0].
> 
> This could be written a LOT more plainly as
> 
>     for ((i=1; $([ -n "$TLC[i]" ]; echo $?) == 0; i++))
> 
> without needing any more characters.  If we're playing golf,
> 
>     for ((i=1; ! $([ "$TLC[i]" ]; echo $?); i++))
> 
> works too.
> 
> Which, by the way, points out another reason you can't just use the
> exit status of a command in a math context:  command success/failure
> are the reverse of true/false when taken as integer values.

Cool! Thanks a lot!

Re: for loop question

[Ray Andrews]

On 11/03/2014 06:43 PM, Bart Schaefer wrote:
> The 0 is appended because [ -n "$TLC[i]" ] does not produce any output,
But in

       if [ -n "$TLC[i]" ]

... the test must surely produce an answer of some sort.  Yes or no.  If 
that answer is not
acceptable as 'arithmetic', then what is it?

    ! $([ -n "$TLC[i]" ]; echo $?)

'echo $?' seems to receive a one or a zero, and that's 'arithmetic', so
why does

     ! $([ -n "$TLC[i]" ]);

... not do exactly the same thing? Why do we need 'echo' to cough up
a testable number?  What does the '$?' do but carry the return  value of
the test? And what would break if the test just returned it's own
return value with no need for help from 'echo $?' to restate it? When we use
the 'if' form above, we don't need the help of 'echo $?' so why do
we need it inside 'for (('? It seems like a pointless limitation.
Maybe a trivial limitation, but still pointless. Indeed:

    [ -n "$TLC[i]" ] && echo "Yup, there's something in there"
   
... so the [] test doesn't even need 'if' to work, it *does* produce
a testable return value, and IMHO that's 'arithmetic', so it should
work in 'for (('.

Which, by the way, points out another reason you can't just use the
exit status of a command in a math context:  command success/failure
are the reverse of true/false when taken as integer values.

Sure, but we get used to using  '!' to reverse true/false, no?  You 
hafta wonder, tho, how that
convention ever got started. '0' is 'false', yet functions return '0' on 
success.  AFAIK it has
always been like that.

Re: for loop question

[Stephane Chazelas]

2014-11-03 18:43:38 -0800, Bart Schaefer:
[...]
> This could be written a LOT more plainly as
> 
>     for ((i=1; $([ -n "$TLC[i]" ]; echo $?) == 0; i++))
[...]

Oh, for some reason, I didn't expect that to work assuming the
$(...) would be expanded only once before the loop starts which
of course was stupid. It was not my intention to obfuscate the
code.

-- 
Stephane

Re: for loop question

[Bart Schaefer]

On Nov 3, 10:37pm, Ray Andrews wrote:
} Subject: Re: for loop question
}
} On 11/03/2014 06:43 PM, Bart Schaefer wrote:
} > The 0 is appended because [ -n "$TLC[i]" ] does not produce any output,
} But in
} 
}        if [ -n "$TLC[i]" ]
} 
} ... the test must surely produce an answer of some sort.

The *nix command paradigm is organized around two concepts:  (1) passing
data through file streams, of which stdin and stdout are the basis for
connecting commands as pipelines; and (2) the input argument array and
integer exit status mapped straight onto C's main() function, with zero
for success and any nonzero value for failure.  The idea is that there
are many ways to fail (among which you might want to distinguish) but
only one way to succeed, and zero is a conviently unique value.

Shells are designed for linking together other programs (some of which
are built in to the shell) using these two paradigms.  Conditions like
"if" and "while" test the exit status; pipelines and substitutions use
the data streams.  If the exit status and the data stream were mixed
together, you could never pass pure data through a pipeline, and you
could never test for program failure without sticking your fingers in
its data stream.

$? [and $status in csh and zsh] exists for the purpose of transforming
an exit status back into an argument list; "echo" [and "print"] is for
transforming argument lists into stream data; "test" transforms an
argument list into an exit status; "read" [and $(...) which originally
was `...`] converts a stream to an argument list.  Plug these four
things together in the right ways and you can build any program linkage
you need.  This is the basic elegance of the shell.

Math operations and all the various parameter manipulations were bolted
onto the top of this model for efficiency, adding complexity for speed
at the cost of elegance.  Originally one had to call programs like
"expr" and "basename" to do all that work, and read back the results
from the their stdout streams.  (More ways to transform argument lists
into stream data.)  But that required forking processes and setting up
data pipes and was slow and expensive, so ...

} Yes or no. If that answer is not acceptable as 'arithmetic', then what
} is it?

It's an exit status, which is a thing separate from the data model.  If
you want it as data (argument list or stream), you have to convert it.
If "test" produced both an exit status and an output stream, you'd have
to be constantly throwing away one or the other, so it only produces
the exit status because that's what you want most often, and you use $?
if you need that in another form.

Shell math works on argument lists (which happen to be interpreted as
variable names and numbers), not on exit status values.  (( )) is just
syntactic sugar for converting numbers to exit status, and $(( )) is
sugar for converting from numbers back to arguments.

I won't quote the rest of your questions, because I think the above has
answered them.

If you REALLY want to understand why shells are the way they are, you
need to study the history of C and UNIX.

Re: for loop question

[Ray Andrews]

On 11/03/2014 11:13 PM, Bart Schaefer wrote:
> } ... the test must surely produce an answer of some sort.
>
> The *nix command paradigm is organized around two concepts:
Thanks Bart, that was very deeply informative, it helps me understand 
the bedrock
on which this is all built.  I asked an invalid question. It is not that 
an exit status is or
isn't 'arithmetic', it is that an exit status is not part of a data 
stream.  It is not *what*
it is, but *where* it is that is relevant. I now see that:

     for ((i=1; ! $([ -n "$TLC[i]" ]; echo $?); i++))

... is simple.