Printing arrays for use with $()

Printing arrays for use with $()

[DervishD]

    Hi all :)

    I have a script which produces a list of files to stdout. It just
prints an array (namely, something like "print -l -- $array" right
now), and I want to use that list in this way:

    whatevercommand $(myscript)

    Obviously the above doesn't work because the spaces in the file
names are not quoted (special character like the square brackets are,
though), but I cannot do this:

    print -l -- ${(qq)array}

    for printing because then even the single quotes are quoted (this
happens too using three and four 'q' flags).

    Is there any way for printing the array to stdout and being able
to use that output as arguments for another command? Please note that
this is a script, not a function, so I cannot return the array, I
must print it :(

    Thanks a lot in advance, dudes :)

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Printing arrays for use with $()

[Bart Schaefer]

On Aug 11,  6:16pm, DervishD wrote:
}
}     Is there any way for printing the array to stdout and being able
} to use that output as arguments for another command?

The short answer is, no.

The long answer is that *something* has to understand the format of the
printed output and convert it back into individual strings.  That means
that either

(1) you need a command that reads from standard input rather than taking
command line arguments, or

(2) you must be in control of the command line so that you can do the
appropriate parsing yourself.

For an example of (2), you could

    print -N -- $array

but then the caller must have IFS=$'\0' to split on the nuls without also
splitting on whitespace.

Re: Printing arrays for use with $()

[DervishD]

    Hi Bart :)

 * Bart Schaefer <schaefer@brasslantern.com> dixit:
> On Aug 11,  6:16pm, DervishD wrote:
> }     Is there any way for printing the array to stdout and being able
> } to use that output as arguments for another command?
> The short answer is, no.

    Oh :(
 
> (1) you need a command that reads from standard input rather than
> taking command line arguments, or

    Possible on some commands only. This list is going to be used in
many commands like 'cp', 'du', 'pax' and even in other scripts.

> (2) you must be in control of the command line so that you can do the
> appropriate parsing yourself.

    Impossible for most commands. Not many commands accept NULL
separated arguments (I can always use xargs, but it's a mess
sometimes).

    So, if I must pass a list of filenames from one app to another in
the most generic way, which are my options? The problem is that the
script that generates the lists must function just the same as the
shell itself when globbing, I mean, it's output must be
indistinguishable from a simple glob. These two command lines must be
equivalent:

    du -s *.whatever
    du -s `myscript`

    If I could return an array from the script, it would do too, but
since it is executed in a subshell, that's impossible, am I wrong?

    Thanks a lot for your fast answer, Bart :)

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Printing arrays for use with $()

[DervishD]

    Hi Bart :)

 * Bart Schaefer <schaefer@brasslantern.com> dixit:
> On Aug 11,  6:16pm, DervishD wrote:
> }     Is there any way for printing the array to stdout and being able
> } to use that output as arguments for another command?
> The short answer is, no.

    I've been trying this (without success):

    command $(printf '"%s "' $array) 

    The thing is that the 'printf' command prints the file names
exactly as I want them, within double quotes, but 'command' see those
double quotes... quoted. Any way of avoiding this? I think that it
may be a solution.

    I'm completely lost and I don't know how to make this script. The
only solution seems to be convert it to a shell function and use
'reply', but I need it to be able to be run as a command, not as a
function :(

    The worst thing is that I had the intention of using a couple
more scripts in this way, generating a list of files to work on and
dumping it to stdout :( but without being able to solve this
problem...

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Printing arrays for use with $()

[Wayne Davison]

On Fri, Aug 12, 2005 at 12:28:47AM +0200, DervishD wrote:
>     The thing is that the 'printf' command prints the file names
> exactly as I want them, within double quotes, but 'command' see those
> double quotes... quoted. Any way of avoiding this?

That's what "eval" is for -- it will re-evaluate the string, and the
quotes will affect how the string is parsed into a command:

    eval "command" $(printf '"%s" ' $array)

..wayne..

Re: Printing arrays for use with $()

[DervishD]

    Hi Wayne :)

 * Wayne Davison <wayned@users.sourceforge.net> dixit:
> On Fri, Aug 12, 2005 at 12:28:47AM +0200, DervishD wrote:
> >     The thing is that the 'printf' command prints the file names
> > exactly as I want them, within double quotes, but 'command' see those
> > double quotes... quoted. Any way of avoiding this?
> That's what "eval" is for -- it will re-evaluate the string, and the
> quotes will affect how the string is parsed into a command:
>     eval "command" $(printf '"%s" ' $array)

    Been there, done that... and it didn't work :? But obviously I
did something wrong, because it DOES work now. Probably I was tired
enough to make a typo and don't seeing it...

    Thanks a lot, Wayne :) It's not the perfect solution but I think
it would do by now.

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Printing arrays for use with $()

[Bart Schaefer]

On Aug 11,  7:01pm, DervishD wrote:
}
}     Hi Bart :)

"Hi, Raúl!"

}     Impossible for most commands.

Hence the short answer ...

}     So, if I must pass a list of filenames from one app to another in
} the most generic way, which are my options?

It all comes back to having some degree of control over the calling
environment.  For example, if it were sufficient that

	du -s *.whatever
and
	eval du -s `myscript`

were equivalent, then

	print -r -- ${(q)array}

should be enough.

On Aug 12, 12:28am, DervishD wrote:
} Subject: Re: Printing arrays for use with $()
}
}     I've been trying this (without success):
} 
}     command $(printf '"%s "' $array) 

Even if that would work (which, as you've seen, it does not) why would
you want to include a trailing space *inside* the double quotes?

}     The worst thing is that I had the intention of using a couple
} more scripts in this way, generating a list of files to work on and
} dumping it to stdout :( but without being able to solve this
} problem...

Well, you could always change tack entirely and instead of

	du -s `myscript`

you could call

	myscript du -s

That is, have your script work the way zargs does.  Or you could modify
zargs to use ${(z)...} on its list of files, and then run

	qzargs -- `myscript` -- du -s

But again all of this assumes you can control the caller, which means
you could just as easily require IFS=$'\0'.  There's just no way around
this, given the semantics of $(...) and word splitting.

Re: Printing arrays for use with $()

[DervishD]

    Hi Bart :)

 * Bart Schaefer <schaefer@brasslantern.com> dixit:
> On Aug 11,  7:01pm, DervishD wrote:
> }     So, if I must pass a list of filenames from one app to another in
> } the most generic way, which are my options?
> It all comes back to having some degree of control over the calling
> environment.  For example, if it were sufficient that
> 
> 	du -s *.whatever
> and
> 	eval du -s `myscript`
> 
> were equivalent, then
> 
> 	print -r -- ${(q)array}
> 
> should be enough.

    Which is not. I mean, it is if I use the eval solution (which, as
I told Wayne, I had tested before without success, probably a typo.
Oh, no, examining my history, the problem was the missing 'q' flag,
that's all).

> }     I've been trying this (without success):
> }     command $(printf '"%s "' $array) 
> Even if that would work (which, as you've seen, it does not) why would
> you want to include a trailing space *inside* the double quotes?

    A typo, my fault O:)
 
> }     The worst thing is that I had the intention of using a couple
> } more scripts in this way, generating a list of files to work on and
> } dumping it to stdout :( but without being able to solve this
> } problem...
> 
> Well, you could always change tack entirely and instead of
> 
> 	du -s `myscript`
> 
> you could call
> 
> 	myscript du -s
> 
> That is, have your script work the way zargs does.

    This is a bit difficult because it forces me to change the way
that the command line is parsed. I mean, the script gets a globbing
pattern or a list of files to process (and produces another list),
and if one of the parameters is a command, I must think about a way
of separating it from the list of files. Something like:

    myscript -c "du -s" files...

    Not much difficult (the script already uses 'getopts') but a bit
messy if I must pass parameters with spaces in them in the command.
It's much easier to use eval, then. Obviously I can use zargs as a
model (using "-- command args"). I think that it could be the way...

> Or you could modify
> zargs to use ${(z)...} on its list of files, and then run
> 
> 	qzargs -- `myscript` -- du -s
> 
> But again all of this assumes you can control the caller, which means
> you could just as easily require IFS=$'\0'.

    Do you mean something like this?:

    IFS=$'\0' du -s `myscript`

    Because it doesn't work, neither (nor exporting IFS with NULL as
its value, zsh still splits on spaces).

    I'm thinking about another solution that could be better, since
sometimes I want to manually review the list before passing it to the
command (and the scripts generates a *different* list each time is
called):

    array=(`myscript args`)

    and making 'myscript' to do a simple 'print -l'. Obviously, the
problem is more or less the same, I need 'eval' or something like
that. Could I do the above, using 'print -N', and after that forcing
the split in NULLs? I've tested this (doesn't work):

    array=(`print -N -- $list`)
    print -l ${(s:$'\0':)array}

    I can use 'eval', of course:

    eval array=\(`print -r -- ${(q)list}`\)

    But it is even more messy to write than using the command itself,
so...  Obviously I'm missing a lot of things here :( but this is the
perfect solution (using an array, I mean) because that allows me to
review the generated list and using it afterwards for more than one
command (which I do frequently, so it will be better than just do
something like "eval command `myscript`"). Any way of doing this
without much mess?

    Thanks a lot for all the help and teaching you're providing :)

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Printing arrays for use with $()

[Bart Schaefer]

On Aug 12, 10:29am, DervishD wrote:
}
} > But again all of this assumes you can control the caller, which means
} > you could just as easily require IFS=$'\0'.
} 
}     Do you mean something like this?:
} 
}     IFS=$'\0' du -s `myscript`

Yes, except that you need a semicolon ...

    IFS=$'\0'; du -s `myscript`

... which also means that you probably need to save and restore the old
value of IFS.  From that standpoint the "eval" is likely better.

}     I'm thinking about another solution that could be better, since
} sometimes I want to manually review the list before passing it to the
} command (and the scripts generates a *different* list each time is
} called):
} 
}     array=(`myscript args`)

A potential way to do this would be to have myscript print the entire
assignment expression:

    print -r -- array=\( ${(q)array} \)

and then have the caller simply do:

    eval $(myscript args)

} Could I do the above, using 'print -N', and after that forcing
} the split in NULLs? I've tested this (doesn't work):
} 
}     array=(`print -N -- $list`)

This doesn't work because in an array context zsh is going to split on
$IFS during the assignment itself, so you already have the wrong thing
in $array before you even get as far as printing it.  You need to use
a scalar assignment:

    notyetarray=`print -N -- $list`
    array=(${(s:$'\0':)notyetarray})

Or (note the double quotes):

    array=( ${(ps:\0:)"$(print -rN -- $list)"} )

} Any way of doing this without much mess?

The above is about as un-messy as it gets.

Re: Printing arrays for use with $()

[DervishD]

    Hi Bart :)

 * Bart Schaefer <schaefer@brasslantern.com> dixit:
> }     I'm thinking about another solution that could be better, since
> } sometimes I want to manually review the list before passing it to the
> } command (and the scripts generates a *different* list each time is
> } called):
> } 
> }     array=(`myscript args`)
> 
> A potential way to do this would be to have myscript print the entire
> assignment expression:
> 
>     print -r -- array=\( ${(q)array} \)
>
> and then have the caller simply do:
> 
>     eval $(myscript args)

    I prefer the solution you post below...
 
> } Could I do the above, using 'print -N', and after that forcing
> } the split in NULLs? I've tested this (doesn't work):
> } 
> }     array=(`print -N -- $list`)
> 
> This doesn't work because in an array context zsh is going to split on
> $IFS during the assignment itself, so you already have the wrong thing
> in $array before you even get as far as printing it.  You need to use
> a scalar assignment:
> 
>     notyetarray=`print -N -- $list`
>     array=(${(s:$'\0':)notyetarray})
> 
> Or (note the double quotes):
> 
>     array=( ${(ps:\0:)"$(print -rN -- $list)"} )

    I've posted a minute ago a possible solution, using more or less
the same mechanism, only I was using ':' as the separator and not
'\0' because it doesn't work for me. I mean, if I do the above, I
cannot use the array directly, I need:

    du -s ${(ps:\0:)array}

    Take a look at my other message about this solution, please, and
make comments to the method I'm using.

    Thanks again, Bart.

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Printing arrays for use with $()

[DervishD]

    Hi Bart :)

 * Bart Schaefer <schaefer@brasslantern.com> dixit:
> That is, have your script work the way zargs does.  Or you could modify
> zargs to use ${(z)...} on its list of files, and then run
> 
> 	qzargs -- `myscript` -- du -s
> 
> But again all of this assumes you can control the caller, which means
> you could just as easily require IFS=$'\0'.  There's just no way around
> this, given the semantics of $(...) and word splitting.

    I've found a possible solution, quick and dirty but it works. The
only problem is that it's a bit messy, but probably you can 'clean'
it a bit O:) It implies an array assignment, but I have no problem
with that because the output is still human readable and using an
array allows me to review the list before using it:

    array=(`print -- ${(j.:.)list}`)
    du -s ${(s.:.)array)

    The "print" part gives us a list with items separated by ':' (I
must find a char that won't be part of the filenames, anyway. ':' is
a very good candidate, but...), and when using we have to undo the
change and do the splitting. It's a bit messy, but at least it works.
There must be a much cleaner solution, but I can't think of one now
:( It's possible to do the above without having to use the split when
using the array? I know I can do this:

    array=(${(s.:.):-`print -- ${(j.:.)list}`})

    equivalent to, in real usage, to:

    array=(${(s.:.):-`myscript args`})

    But I would like to have a simpler assignment...

    Thanks a lot in advance :)

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Printing arrays for use with $()

[DervishD]

    Hi Bart :)

    Sorry for self replying...

 * DervishD <zsh@dervishd.net> dixit:
>     array=(${(s.:.):-`print -- ${(j.:.)list}`})

    Better:

    array=( ${(ps:\0:)"$(print -N -- $list)"} )

    Although I don't know why I need the double quotes if I use ":-"
after the (ps) flag :? Is to avoid the shell splitting the entire
string in the assignment before we can do the (ps)?

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Printing arrays for use with $()

[Bart Schaefer]

On Aug 12,  4:52pm, DervishD wrote:
}
} >     array=( ${(ps:\0:)"$(print -rN -- $list)"} )
} 
}     I've posted a minute ago a possible solution, using more or less
} the same mechanism, only I was using ':' as the separator and not
} '\0' because it doesn't work for me.

I gather from your subsequent posting that you got it to work after all.

On Aug 12,  4:57pm, DervishD wrote:
}
}     array=( ${(ps:\0:)"$(print -N -- $list)"} )

I still recommend "print -rN" here.

}     Although I don't know why I need the double quotes if I use ":-"
} after the (ps) flag :? Is to avoid the shell splitting the entire
} string in the assignment before we can do the (ps)?

The results of $(...) and `...` are always subject to word splitting
unless quoted.  It's part of the definition of those substitutions.  
So, yes.

Re: Printing arrays for use with $()

[DervishD]

    Hi Bart :)

 * Bart Schaefer <schaefer@brasslantern.com> dixit:
> On Aug 12,  4:57pm, DervishD wrote:
> }     array=( ${(ps:\0:)"$(print -N -- $list)"} )
> I still recommend "print -rN" here.

    Why? Just in case some file has a '\' sequence in its name, a
quote or something like that?

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Printing arrays for use with $()

[Bart Schaefer]

On Aug 13, 10:28am, DervishD wrote:
} > I still recommend "print -rN" here.
} 
}     Why? Just in case some file has a '\' sequence in its name, a
} quote or something like that?

Yep.

Re: Printing arrays for use with $()

[DervishD]

    Hi Bart :)

 * Bart Schaefer <schaefer@brasslantern.com> dixit:
> On Aug 13, 10:28am, DervishD wrote:
> } > I still recommend "print -rN" here.
> }     Why? Just in case some file has a '\' sequence in its name, a
> } quote or something like that?
> Yep.

    OK. I finally used this solution, but added an option to the
script so I can use '\0' or '\n' as the separator, and after that
I've setup an alias:

    myalias='reply=( ${(f)"$(script)"} ) ; print -l $reply'

    This way I can use 'reply' for storing the list of files when the
shell is interactive, and if I use 'script' within another script, I
just write the above. I can use the script with the option of using
NULL's as separator and pipe the output thru 'xargs' or the like.

    Thanks a lot for your invaluable help, Bart :))

    In case anyone is interested, the script shuffles a list of MP3
files and chooses a set to fit some given size (for example, the
space available in your MP3 player, a 74minutes CD, etc. I had it
implemented as a couple of shell functions, but I needed to make it
an script, that's why I made so much noise in the list.

    I'm going to release under GPL this and a few other scripts in
some weeks, just in case someone wants to take a look. There is very
good code in the scripts, but almost all of them came from the mind
of Bart, so please bother him with the acknowledgements XDD

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Printing arrays for use with $()

[Dan Nelson]

In the last episode (Aug 11), DervishD said:
>     I have a script which produces a list of files to stdout. It just
> prints an array (namely, something like "print -l -- $array" right
> now), and I want to use that list in this way:
> 
>     whatevercommand $(myscript)
> 
>     Obviously the above doesn't work because the spaces in the file
> names are not quoted (special character like the square brackets are,
> though), but I cannot do this:
> 
>     print -l -- ${(qq)array}
> 
>     for printing because then even the single quotes are quoted (this
> happens too using three and four 'q' flags).

Why not use ${(q)array}?  You probably also want to use print -r,
othersize it will try and parse \-escape codes.

dan% a=("file name" single\'quote double\"quote)
dan% echo ${(q)a}
file\ name single\'quote double\"quote
dan% echo ${(qq)a}
'file name' 'single'\''quote' 'double"quote'
dan% echo ${(qqq)a}
"file name" "single'quote" "double\"quote"
dan% print -l -- ${(qq)a}
'file name'
'single'''quote'
'double"quote'
dan% print -rl -- ${(qq)a}
'file name'
'single'\''quote'
'double"quote'

-- 
	Dan Nelson
	dnelson@allantgroup.com

Re: Printing arrays for use with $()

[DervishD]

    Hi Dan :)

 * Dan Nelson <dnelson@allantgroup.com> dixit:
> Why not use ${(q)array}?  You probably also want to use print -r,
> othersize it will try and parse \-escape codes.
> 
> dan% a=("file name" single\'quote double\"quote)
> dan% echo ${(q)a}
> file\ name single\'quote double\"quote
> dan% echo ${(qq)a}
> 'file name' 'single'\''quote' 'double"quote'
> dan% echo ${(qqq)a}
> "file name" "single'quote" "double\"quote"
> dan% print -l -- ${(qq)a}
> 'file name'
> 'single'''quote'
> 'double"quote'
> dan% print -rl -- ${(qq)a}
> 'file name'
> 'single'\''quote'
> 'double"quote'

    Try, with your example, this:

    du $(print -lr -- ${(q)a})

    or any other of the examples ;) They will fail with exactly the
same problem I'm currently having. The problem is that the "print"
command doesn't quote properly its output (obviously, it has to
*print* it, not quote it to make it munchable for other commands).

    Thanks anyway :)

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...