zsh-users
 help / color / mirror / code / Atom feed
* Efficient way to map a list of values to multiple processes, then accumulate their output
@ 2021-11-06 13:40 Zach Riggle
  2021-11-06 16:18 ` Bart Schaefer
  0 siblings, 1 reply; 6+ messages in thread
From: Zach Riggle @ 2021-11-06 13:40 UTC (permalink / raw)
  To: Zsh Users

[-- Attachment #1: Type: text/plain, Size: 1269 bytes --]

Hello all!  Thanks in advance for the assistance.

Ultimately, I'd like to have some quasi-implementation of xargs -P in pure
zsh -- which maps a __function__ to a list of arguments, and returns the
results in an array or associative array.  xargs cannot be used to run
shell functions, so this is out.

I think this is a situation that Zsh is not well-suited for, but let's say
I have a list of files that I want to perform some processing on, then
gather the results when the processing has completed.

Simply backgrounding each process with & and then wait'ing on all jobs to
complete should work, in theory.  However, the output from each invocation
may be intermixed, even in the simple case of "one line of output per file"
the order of completion is non-deterministic and the output may become
garbled.

This seems something that would fit nicely with co-processes, but it
appears each zsh instance can only have one.

Another idea is to farm the output of each to a temporary file, and have an
associative array of e.g. [input]=$(mktemp) as well as [input]=job_id and
then grabbing the output as each job completes.

I expect I am not the first person to try this (equivalent to the Python
multiprocessing.map) but I figured it was worth asking!

*Zach Riggle*

[-- Attachment #2: Type: text/html, Size: 1657 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2021-11-08 20:46 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-06 13:40 Efficient way to map a list of values to multiple processes, then accumulate their output Zach Riggle
2021-11-06 16:18 ` Bart Schaefer
2021-11-06 21:58   ` Zach Riggle
2021-11-06 22:55     ` Lawrence Velázquez
2021-11-08 20:27       ` Zach Riggle
2021-11-08 20:45         ` Zach Riggle

Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/zsh/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).