Efficient way to map a list of values to multiple processes, then accumulate their output

[Zach Riggle <zachriggle@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1269 bytes --]

Hello all!  Thanks in advance for the assistance.

Ultimately, I'd like to have some quasi-implementation of xargs -P in pure
zsh -- which maps a __function__ to a list of arguments, and returns the
results in an array or associative array.  xargs cannot be used to run
shell functions, so this is out.

I think this is a situation that Zsh is not well-suited for, but let's say
I have a list of files that I want to perform some processing on, then
gather the results when the processing has completed.

Simply backgrounding each process with & and then wait'ing on all jobs to
complete should work, in theory.  However, the output from each invocation
may be intermixed, even in the simple case of "one line of output per file"
the order of completion is non-deterministic and the output may become
garbled.

This seems something that would fit nicely with co-processes, but it
appears each zsh instance can only have one.

Another idea is to farm the output of each to a temporary file, and have an
associative array of e.g. [input]=$(mktemp) as well as [input]=job_id and
then grabbing the output as each job completes.

I expect I am not the first person to try this (equivalent to the Python
multiprocessing.map) but I figured it was worth asking!

*Zach Riggle*

[-- Attachment #2: Type: text/html, Size: 1657 bytes --]