zargs parallelization support blows up with |&

zargs parallelization support blows up with |&

[Dima Kogan]

Hi. I can use this to invoke a perl script 10 times, 2 processes at a
time:

 zargs -P2 -i -l1 `seq 10` -- perl -e 'sleep 20' {}

This works ok. However, when I do the same thing, but filter stdout and
stderr for some output, then ALL the processes are invoked at the same
time, not just 2 at a time like I asked:

 zargs -P2 -i -l1 `seq 10` -- perl -e 'sleep 20' {} |& grep x

I don't know if this is a bug or an expected consequence of some
internal implementation details, but this is a really nasty failure
mode.

Thanks for looking.
dima

Re: zargs parallelization support blows up with |&

[Bart Schaefer]

On Jul 27,  2:36am, Dima Kogan wrote:
}
} [... when piping] stdout and
} stderr for some output, then ALL the processes are invoked at the same
} time, not just 2 at a time like I asked:
} 
}  zargs -P2 -i -l1 `seq 10` -- perl -e 'sleep 20' {} |& grep x
} 
} I don't know if this is a bug or an expected consequence of some
} internal implementation details, but this is a really nasty failure
} mode.

The problem is that when you append a pipe to the end, zargs ends up
as a subshell, and you can't do job control in a subshell (which you
would have seen if you weren't grepping stderr):

(eval):wait:1: can't manipulate jobs in subshell

The workaround for now is as follows:

Copy the zargs script file to somewhere in your $path, e.g. ~/bin/.

Edit the copy to add a #!/bin/zsh (or your local path) at the top,
and "chmod +x" the file.

Then either always run zargs as a script instead of an autoload, or
when you need to pipeline it, use "command zargs".  This is a little
less convenient because it loses access to interactively-defined
functions, but if you're zargs-ing another external command such as
perl it should work fine.

Re: zargs parallelization support blows up with |&

[Bart Schaefer]

On Jul 27,  9:15am, Bart Schaefer wrote:
}
} The problem is that when you append a pipe to the end, zargs ends up
} as a subshell, and you can't do job control in a subshell (which you
} would have seen if you weren't grepping stderr):
} 
} (eval):wait:1: can't manipulate jobs in subshell

This turned out to be easier to fix than I expected:

diff --git a/Functions/Misc/zargs b/Functions/Misc/zargs
index 71fd428..28ebca7 100644
--- a/Functions/Misc/zargs
+++ b/Functions/Misc/zargs
@@ -73,7 +73,7 @@
 emulate -L zsh || return 1
 local -a opts eof n s l P i
 
-local ZARGS_VERSION="1.4"
+local ZARGS_VERSION="1.5"
 
 if zparseopts -a opts -D -- \
 	-eof::=eof e::=eof \
@@ -254,7 +254,7 @@ then
     bg='&'
     if zmodload -i zsh/parameter 2>/dev/null
     then
-	wait='wait %${(k)^jobstates[(R)running:*]}'
+	wait='wait ${${jobstates[(R)running:*]/#*:/}/%=*/}'
     else
 	wait='wait'
     fi

Re: zargs parallelization support blows up with |&

[Dima Kogan]

Bart Schaefer <schaefer@brasslantern.com> writes:

> On Jul 27,  9:15am, Bart Schaefer wrote:
> }
> } The problem is that when you append a pipe to the end, zargs ends up
> } as a subshell, and you can't do job control in a subshell (which you
> } would have seen if you weren't grepping stderr):
> } 
> } (eval):wait:1: can't manipulate jobs in subshell
>
> This turned out to be easier to fix than I expected:
>
> <patch>

The patch works well for me. Thanks!