xargs with zsh function

xargs with zsh function

[Ray Andrews]

Can it be done?

     $ ... | xargs my_function

it seems that xargs only likes binary commands.  There was some thought 
on the web that something roughly like:

     $ ... | xargs "zsh -c my_function"

might work but 'zsh -c' seems to be a virginal shell that doesn't know 
my_function exists.  How do we pipe into a function?

     $ func_2 "${(f)$(func_1 *)}"

... works but there's the usual travail getting zsh to break on lines.  
Mind that's always the case so perhaps that's as good as it gets.  But I 
figure that piping should be doable.

Re: xargs with zsh function

[Bart Schaefer]

On Sun, Jan 17, 2021 at 3:12 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> Can it be done?

The problem isn't the pipe, it's that xargs isn't zsh.

>  How do we pipe into a function?

That's the wrong question.  You're not piping into a function, you're
piping into a process (xargs) that turns its input lines into a set of
command-line arguments.

How about this:

$ autoload zargs
$ xzargs() { zargs -- "${(f)$(read -d '' -E)}" -- "$@" }
$ ... | xzargs my_function

That does use a lot of memory, reading the entire stdin and turning it
into a command line.

What you really want to do is write a nested shell loop that consumes
N lines, then runs my_function, then does that again until it runs out
of input; and pipe to that loop.  There are a bunch of ways to do
that.  E.g.:

... | while (true)
  do
    set --
    for i in {1..10}
    do
      read $i || break
    done
    my_function "$@"
    [[ $# -eq 10 ]] || break
  done

Re: xargs with zsh function

[Ray Andrews]

On 2021-01-17 7:10 p.m., Bart Schaefer wrote:
>
> How about this:
>
> $ autoload zargs
> $ xzargs() { zargs -- "${(f)$(read -d '' -E)}" -- "$@" }
> $ ... | xzargs my_function
>
>
Playing with that running this script (test):

    autoload zargs
    xzargs() { zargs -t -- "${(f)$(read -d '' -E)}" -- "$@" }

    function el ()
    {
    echo howdy
    }

    vvar=( 'mnt' 'rap' )

    print -l "$vvar" | xzargs el

------------------------------------------
$ . test
xzargs: bad math expression: operator expected at `rap'
el mnt rap
howdy

... It works fine with a single literal input or a single string 
variable but I'd want to be throwing large numbers of lines at this.  I 
know the receiving function might have to be feed them one at a time, 
which is easy, but I can't get zargs to swallow more than one string.

Re: xargs with zsh function

[Bart Schaefer]

On Mon, Jan 18, 2021 at 2:14 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> Playing with that running this script (test):
>
> $ . test
> xzargs: bad math expression: operator expected at `rap'

There's something in your test script you're not showing us; nothing in

xzargs() { zargs -t -- "${(f)$(read -d '' -E)}" -- "$@" }

would be evaluating a math expression.

> ... It works fine with a single literal input or a single string
> variable but I'd want to be throwing large numbers of lines at this.

The way xargs works is to gather up N lines until a "large enough"
argument list is assembled, and then call the command with those
arguments.  The way zargs works is to take all of those arguments on a
single command line and break them up into batches of N.

print -l "$vvar"  # this only prints one line, argument will be "mnt rap"
print -l $vvar    # This prints two lines, arguments will be "mnt" and "rap"

If you actually want to see this in action you need to specify that
zargs should only process one argument at a time.

% xzargs() { zargs -n 1 -t -- "${(f)$(read -d '' -E)}" -- "$@" }
% print -l $vvar | xzargs el
el mnt
howdy
el rap
howdy

Re: xargs with zsh function

[Ray Andrews]

On 2021-01-18 3:09 p.m., Bart Schaefer wrote:
> On Mon, Jan 18, 2021 at 2:14 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>>
>> There's something in your test script you're not showing us;
nothing in
> xzargs() { zargs -t -- "${(f)$(read -d '' -E)}" -- "$@" }
>
> would be evaluating a math expression.
Nope that was the entire thing.  Some artifact somewhere?  Some ... 
dunno.  Something weird with the terminal perhaps, I restarted and math 
issues are gone.  Very strange.  By now I should remember to always try 
things in a clean shell and fresh terminal before reporting some difficulty.
>
> % xzargs() { zargs -n 1 -t -- "${(f)$(read -d '' -E)}" -- "$@" }
>
Thanks.

Re: xargs with zsh function

[Ray Andrews]

On 2021-01-18 9:18 p.m., Ray Andrews wrote:

I can't find any documentation of zargs which might help but behavior 
seems mysterious:

#!/usr/bin/zsh

autoload zargs
xzargs() { zargs  -n1 -t -- "${(f)$(read -d '' -E)}" -- "$@" }

typeset -g xinput=( 'mnt' 'rap' )

echo '\nfirst:'
eval "ls (#i)$xinput[1]"
eval "ls $xinput[2]"

function el ()
{
     echo '\nin function:'
     eval "ls (#i)$xinput[1]"
     eval "ls $xinput[2]"
}

echo '\nsecond:'
eval "ls (#i)$xinput[1]"
eval "ls $xinput[2]"

print "$xinput" | xzargs el

echo '\nthird:'
eval "ls (#i)$xinput[1]"
eval "ls $xinput[2]"

-----------------------------------------------
Output:
My system as she lies:

$ . test

first:
mnt
rap

second:
mnt
rap
el mnt rap

in function:
(eval):1: no matches found: (#i)mnt
rap

third:
mnt
rap

-----------------------------------------------
Now from clean shell:

h5--9-Debian1#  . test

first:
(eval):1: no matches found: (#i)mnt
rap

second:
(eval):1: no matches found: (#i)mnt
rap
el mnt rap

in function:
(eval):1: no matches found: (#i)mnt
rap

third:
(eval):1: no matches found: (#i)mnt
rap

-----------------------------------------------
Perhaps this has something to do with the call in 'el' being to a clean 
shell? Same sort of problem if I do this:

$ ls (#i)c
C  c

$ zsh -f
h5--9-Debian1# ls (#i)c
zsh: no matches found: (#i)c

...  But that's bedrock syntax no?  How can the clean shell not swallow 
it?  Anyway I'd like zargs to work within my environment as it is.

Re: xargs with zsh function

[Lawrence Velázquez]

> On Jan 19, 2021, at 9:30 AM, Ray Andrews <rayandrews@eastlink.ca> wrote:
> 
> Perhaps this has something to do with the call in 'el' being to a clean shell? Same sort of problem if I do this:
> 
> $ ls (#i)c
> C  c
> 
> $ zsh -f
> h5--9-Debian1# ls (#i)c
> zsh: no matches found: (#i)c
> 
> ...  But that's bedrock syntax no?  How can the clean shell not swallow it?

Globbing flags require EXTENDED_GLOB, which is not enabled by default.

> Anyway I'd like zargs to work within my environment as it is.

To establish a known configuration, zargs resets nearly all zsh options for the duration of its execution. Your function should do likewise.

f() {
    emulate -L zsh
    setopt EXTENDED_GLOB
    [also any other options required]

    [...]
}

This ensures a known configuration that is isolated from that of the calling context.

vq
Sent from my iPhone

Re: xargs with zsh function

[Ray Andrews]

On 2021-01-19 9:52 a.m., Lawrence Velázquez wrote:
>> ...  But that's bedrock syntax no?  How can the clean shell not swallow it?
> Globbing flags require EXTENDED_GLOB, which is not enabled by default.
Right you are.
> To establish a known configuration, zargs resets nearly all zsh options for the duration of its execution. Your function should do likewise.
My strong preference would be that zargs be agnostic as to one's 
environment and functions called by it would behave exactly as 
otherwise.  Is the code for zargs a discrete thing?  If so I'm tempted 
to clone it to another name and see if I can make it transparent to 
existing environment.  I'd like to be able to link my functions together 
via piping+zargs.  Mind, I'm imagining that would be a simplification 
but perhaps exactly the opposite.  Besides, piping is so mysterious, 
there's an invisible river of data and it would be interesting to see 
how one gets one's grubby hands on it. Thanks Lawrence I spent hours 
wondering what could possibly be going wrong there.

Re: xargs with zsh function

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

On Tue, Jan 19, 2021 at 9:52 AM Lawrence Velázquez <vq@larryv.me> wrote:
>
> > On Jan 19, 2021, at 9:30 AM, Ray Andrews <rayandrews@eastlink.ca> wrote:
> >
> > Anyway I'd like zargs to work within my environment as it is.
>
> To establish a known configuration, zargs resets nearly all zsh options for the duration of its execution. Your function should do likewise.

The latter is always good practice, but it does raise the point that
zargs is less useful if you aren't using something close to zsh mode.

I'm still pondering if there's a better way to do this, but in the meantime ...

(attached to avoid gmail line wrapping)

[-- Attachment #2: zargs_patch.txt --]
[-- Type: text/plain, Size: 758 bytes --]

diff --git a/Functions/Misc/zargs b/Functions/Misc/zargs
index 28ebca78f..49c177fb6 100644
--- a/Functions/Misc/zargs
+++ b/Functions/Misc/zargs
@@ -70,6 +70,15 @@
 #   this behavior, as do -l/-L, but when -i/-I appear anywhere then -l/-L
 #   are ignored (forced to 1).
 
+# First, capture the current setopts as "sticky emulation"
+if zmodload zsh/param
+then
+  emulate zsh -c "_zarun() { options=( ${(j: :kv)options[@]} )"' eval "$@" }'
+else
+  # Warning?
+  _zarun() { eval "$@" }
+fi
+
 emulate -L zsh || return 1
 local -a opts eof n s l P i
 
@@ -186,7 +195,7 @@ local execute='
     elif (( $opts[(I)-(-verbose|t)] ))
     then print -u2 -r -- "$call"
     fi
-    eval "{
+    _zarun "{
        \"\${(@)call}\"
     } $bg"'
 local ret=0 analyze='

Re: xargs with zsh function

[Ray Andrews]

On 2021-01-19 12:31 p.m., Bart Schaefer wrote:
> On Tue, Jan 19, 2021 at 9:52 AM Lawrence Velázquez <vq@larryv.me> wrote:
> I'm still pondering if there's a better way to do this, but in the 
> meantime ...
> (attached to avoid gmail line wrapping)
It sure would be friendly to have the option of making it transparent.

Re: xargs with zsh function

[Bart Schaefer]

On Tue, Jan 19, 2021 at 12:27 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> My strong preference would be that zargs be agnostic as to one's
> environment and functions called by it would behave exactly as
> otherwise.

That's what the zargs_patch.txt in my other message is supposed to do.

> I'd like to be able to link my functions together
> via piping+zargs.

Pipelines are for linking together things that expect to read standard
input and write standard output.

foo | bar

is the same as

foo > file
bar < file

except that the file is being simultaneously written and read, which
means that at least one of foo or bar has to run in the background.
In zsh, that's "foo", in bash it's "bar", and in a true POSIX shell
it's both of them.  (A patch to cause zsh to work that last way in
shell emulation was recently posted.)

Functions can be written to read standard input and write standard
output, of course, but often they are intended to do work in the
current shell environment.  If you rejigger them into a pipeline, you
force some of them into background subshells, and those are no longer
able to affect the current shell.  That may or may not be OK, but you
should think about it.

Re: xargs with zsh function

[Ray Andrews]

On 2021-01-19 1:39 p.m., Bart Schaefer wrote:
> On Tue, Jan 19, 2021 at 12:27 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>> My strong preference would be that zargs be agnostic as to one's
>> environment and functions called by it would behave exactly as
>> otherwise.
> That's what the zargs_patch.txt in my other message is supposed to do.
>
>> I'd like to be able to link my functions together
>> via piping+zargs.
> Pipelines are for linking together things that expect to read standard
> input and write standard output.
>
> foo | bar
>
> is the same as
>
> foo > file
> bar < file
>
> except that the file is being simultaneously written and read
The above is much easier to understand and of course obviously simpler.  
As I said,
the whole thing is a bit of a mystery.  A lot of stuff attaches to ' | ' 
obviously.

> If you rejigger them into a pipeline, you
> force some of them into background subshells, and those are no longer
> able to affect the current shell.  That may or may not be OK, but you
> should think about it.
Well again I have no idea what the intimacies of it are but at least the 
notion of being able to
link functions analogously to what xargs does is desirable.  If you have 
a patch that does that for
zargs I'd sure like to have it.

Re: xargs with zsh function

[Ray Andrews]

On 2021-01-19 1:39 p.m., Bart Schaefer wrote:
>
> That's what the zargs_patch.txt in my other message is supposed to do.
>
How can I apply that?  I know you guys a diffing and giting all the time 
so you only need to transmit differences but ... well, I do remember 
doing that years ago but I've totally forgotten.  I'll do as you say 
absolutely by wrote.

Re: xargs with zsh function

[Daniel Shahaf]

Bart Schaefer wrote on Sun, Jan 17, 2021 at 19:10:10 -0800:
> What you really want to do is write a nested shell loop that consumes
> N lines, then runs my_function, then does that again until it runs out
> of input; and pipe to that loop.  There are a bunch of ways to do
> that.  E.g.:
> 
> ... | while (true)
>   do
>     set --
>     for i in {1..10}
>     do
>       read $i || break
>     done
>     my_function "$@"
>     [[ $# -eq 10 ]] || break
>   done

If it's fine to call the function with the arguments one at a time, then the
following should do:

    «| xargs -n1 | while IFS= read -r line; do f "${line}"; done»

Re: xargs with zsh function

[Ray Andrews]

On 2021-01-19 5:05 p.m., Ray Andrews wrote:
> On 2021-01-19 1:39 p.m., Bart Schaefer wrote:
>>
>> That's what the zargs_patch.txt in my other message is supposed to do.
>>
> How can I apply that?  I know you guys a diffing and giting all the 
> time so you only need to transmit differences but ... well, I do 
> remember doing that years ago but I've totally forgotten.  I'll do as 
> you say absolutely by wrote.
>
>
Not knowing diff, I copied Bart's changes in by hand and got:

     failed to load module: zsh/param

Editing to change: "zmodload zsh/param" to "zmodload zs/parameter" 
(that's what it seems to be named here, I have nothing named 'param' ), 
former error vanishes and now:

     _zarun: can't change option: zle

Re: xargs with zsh function

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

> Not knowing diff, I copied Bart's changes in by hand and got:
>
>      failed to load module: zsh/param

Yeah, I pasted the wrong diff into the file I then attached.  Sorry about that.

In the meantime I've made some improvements, including correcting this:

>      _zarun: can't change option: zle

It should now preserve option settings as much as possible.  However,
the function itself can't autoload with "sh" emulation in effect; if
you want to use this where zsh is acting as /bin/sh, you need to use:
  emulate zsh -c 'autoload zargs'

Since Ray asked:  Apply this with
 patch -p1 Functions/Misc/zargs < zargs_patch2.txt

[-- Attachment #2: zargs_patch2.txt --]
[-- Type: text/plain, Size: 873 bytes --]

diff --git a/Functions/Misc/zargs b/Functions/Misc/zargs
index 28ebca78f..ecd69f7e4 100644
--- a/Functions/Misc/zargs
+++ b/Functions/Misc/zargs
@@ -70,6 +70,19 @@
 #   this behavior, as do -l/-L, but when -i/-I appear anywhere then -l/-L
 #   are ignored (forced to 1).
 
+# First, capture the current setopts as "sticky emulation"
+if zmodload zsh/parameter
+then
+  emulate $(emulate -l) -c "\
+    _zarun() {
+      options=( ${(j: :kv)options[@]} monitor off zle off )"'
+      eval "$@"
+    }'
+else
+  # Warning?
+  emulate $(emulate -l) -c '_zarun() { eval "$@" }'
+fi
+
 emulate -L zsh || return 1
 local -a opts eof n s l P i
 
@@ -186,8 +199,8 @@ local execute='
     elif (( $opts[(I)-(-verbose|t)] ))
     then print -u2 -r -- "$call"
     fi
-    eval "{
-	\"\${(@)call}\"
+    _zarun "{
+	\"\${call[@]}\"
     } $bg"'
 local ret=0 analyze='
     case $? in

Re: xargs with zsh function

[Ray Andrews]

On 2021-01-20 9:52 p.m., Bart Schaefer wrote:
>
> Since Ray asked:  Apply this with
>   patch -p1 Functions/Misc/zargs < zargs_patch2.txt
Seems to pass my posted test with flying colors.  One thing tho, the 
paths:  My working 'zargs' is:

..../Zsh-5.8/share/zsh/5.8/functions/zargs

I have no directory 'Functions' or 'Misc' anywhere on my system. How is 
it that my zsh directory tree is different?

BTW, speaking of passing data from one function to another supposing we 
have a list of filenames in a file:

filename1
filename2
filename with spaces in it
filename with funny characters in it

... you can feed the lines to 'ls' one at a time using zargs as you've 
shown but 'ls' accepts:

$ ls filename1 filename2 'filename with spaces in it'

... how can the contents of the file be fed to 'ls' as one call, but 
with the spaces protected?  The individual lines need no quotes to 
protect them, but once morphed into a single line the quotes must be 
there.  Can it be done?

Re: xargs with zsh function

[Bart Schaefer]

On Thu, Jan 21, 2021 at 11:56 AM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> On 2021-01-20 9:52 p.m., Bart Schaefer wrote:
> >
> > Since Ray asked:  Apply this with
> >   patch -p1 Functions/Misc/zargs < zargs_patch2.txt
> Seems to pass my posted test with flying colors.  One thing tho, the
> paths:  My working 'zargs' is:
>
> ..../Zsh-5.8/share/zsh/5.8/functions/zargs
>
> I have no directory 'Functions' or 'Misc' anywhere on my system. How is
> it that my zsh directory tree is different?

My instruction was for patching the zsh source tree, not an installed location.

> BTW, speaking of passing data from one function to another supposing we
> have a list of filenames in a file:
>
> ... how can the contents of the file be fed to 'ls' as one call, but
> with the spaces protected?

That should "just work".  You don't usually even need zargs for that:

% ls -dl -- ${(f)"$(<filenames.txt)"}

The reason zargs exists is in case the above "ls" complains "argument
list too long".  In that case you can do

% zargs -- ${(f)"$(<filenames.txt)"} -- ls -dl

to break up the argument list into slices that "ls" will accept.

Re: xargs with zsh function

[Ray Andrews]

On 2021-01-21 1:50 p.m., Bart Schaefer wrote:


> My instruction was for patching the zsh source tree, not an installed location.
Ah.  Well it worked patched into my copy.  I trust I've not busted 
something.
>
> That should "just work".  You don't usually even need zargs for that:
>
> % ls -dl -- ${(f)"$(<filenames.txt)"}
... and to the point, I can swap one of my functions for 'ls' and it 
seems  seamless.
>
> The reason zargs exists is in case the above "ls" complains "argument
> list too long".  In that case you can do
>
> % zargs -- ${(f)"$(<filenames.txt)"} -- ls -dl
>
> to break up the argument list into slices that "ls" will accept.
You know, one sentence there and I finally get it.  Why don't they 
tellya these things?  All good, thanks Bart.