From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 8192 invoked from network); 6 Nov 2021 13:41:55 -0000 Received: from zero.zsh.org (2a02:898:31:0:48:4558:7a:7368) by inbox.vuxu.org with ESMTPUTF8; 6 Nov 2021 13:41:55 -0000 ARC-Seal: i=1; cv=none; a=rsa-sha256; d=zsh.org; s=rsa-20210803; t=1636206115; b=AcEa5zGdZlwo+fMvAbcHW2tU/cq/PaIAPrSgGX/4eCdQiaEg4tSTGqvhc2VxZ+sWQ4Mzpcm9bf ofjDiVR2ezjmCON9Pf1brugcTMqm5gpRL4WtF1HOONi5/Y+SNfiMdjaK+1aGopUUlVTU2ZTr/i MscKAXfgFPBdC0ewt+rtgczJS1yge0Mxwp5PBgPUiCUn1A6Jy7/CT5NWY1RKgLl5phZowj51RB JgBJcPTr0Fy/C/G2cwuTtOrOYfPhAoIjnWB28zZRuazq04LU9HCAT2IGq9QaG981Ajs2onOTNa /AOEBQwAEQlDagFpZPKKoGahRYvvyUqRm30vFsR0lTvU8A==; ARC-Authentication-Results: i=1; zsh.org; iprev=pass (mail-wr1-f44.google.com) smtp.remote-ip=209.85.221.44; dkim=pass header.d=gmail.com header.s=20210112 header.a=rsa-sha256; dmarc=pass header.from=gmail.com; arc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed; d=zsh.org; s=rsa-20210803; t=1636206115; bh=iVeEFR3TgHHEl6pleO1mf4ewGxGVcl3MoubAI3D+rfA=; h=List-Archive:List-Owner:List-Post:List-Unsubscribe:List-Subscribe:List-Help: List-Id:Sender:Content-Type:To:Subject:Message-ID:Date:From:MIME-Version: DKIM-Signature:DKIM-Signature; b=T0o5Q1rbyqNE5BZPJVhz+uf9HPNq3fQh37BGZQ0+0K9UEYeHeEAZMzFhXG54+3XY0/HJqL6qvi cnT7m1e8vCe8kfq6lqKk5duuPKDs55eMEWO5CSK0yl+0bIFoggJ4FNaZ2GfRRBKg/4tyFW+TFY GYQha08Nw2MWlKF+psbObME9idWt1Ca0ZJypv/+vVmtFL/Oke2xbInw1KfCsXHnSxGAKw8quIw cDn4xkyfDGEQSAkcZCamL/fMRFdiEVuan0CUUSxTQTFtC7NudRCsTK5K0iZCkqNBK4tzq/Fk6n MGc7co1e9UmNmwf+pDP1LggUMeYnAok9gCcslwOEt80FlQ==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zsh.org; s=rsa-20210803; h=List-Archive:List-Owner:List-Post:List-Unsubscribe: List-Subscribe:List-Help:List-Id:Sender:Content-Type:To:Subject:Message-ID: Date:From:MIME-Version:Reply-To:Cc:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References; bh=bBgNab7REdVl+SviiK+uiL0gPCzgPViae4/JyPmHOt8=; b=mgx1fhD0leh5htV8mpZtNj5HJY 4Z8+L4YB5jP6iohU7KZCT9eW9RH/WR5P+tqCqDT7ZzZ8WZXae6g2h3GGMzmkv84dQhqw8XJbzwWZ5 SjYjH0Xo3wDYRV9nGcZWP6v9ZVzrUfSuW1GQ1XZqY3+MVuDa31iK63Nzb7yFE7u35RFMylfY6+TfV ie8s1RDQCYzx2iwHD8abNYeFm/wSt1B6FGEtBg8Im3vpZIBLWjcN9r3tHnvz6kquiJrXo1RlyyvT0 GsRmuUXUUJWQPXaQomANY/3pGiRKxyNIt45HatvKNoY9iAjYHrZJv3ozCTI7Tp8EmEgebPQa0Im5R CQgkT3ZQ==; Received: from authenticated user by zero.zsh.org with local id 1mjLxB-000JX9-Py; Sat, 06 Nov 2021 13:41:53 +0000 Authentication-Results: zsh.org; iprev=pass (mail-wr1-f44.google.com) smtp.remote-ip=209.85.221.44; dkim=pass header.d=gmail.com header.s=20210112 header.a=rsa-sha256; dmarc=pass header.from=gmail.com; arc=none Received: from mail-wr1-f44.google.com ([209.85.221.44]:38910) by zero.zsh.org with esmtps (TLS1.3:TLS_AES_128_GCM_SHA256:128) id 1mjLwN-000Iqz-O7; Sat, 06 Nov 2021 13:41:04 +0000 Received: by mail-wr1-f44.google.com with SMTP id u18so18202867wrg.5 for ; Sat, 06 Nov 2021 06:41:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=bBgNab7REdVl+SviiK+uiL0gPCzgPViae4/JyPmHOt8=; b=MragX9qEdVwd28MIubCn+CEudIdCJlyjow9pxaTcEaTUtUUZqUFciOGT86kFSnIsQ2 1gsCWS0BPGXS23qAjSi04NCu2VYqfSb1IlvZUue3YNbjVWoHvUCH0xRDDV5J8G0YxqkW TaVeVfeC9TrfCWftEQR39wLOa+I9a26Esja2Gj03fDOGjQ/qP1UeVb+7u7TbDCOIA8nG ieAvADffEif8vfRlR9zPtVzv8VCObwSPaqsJXWEP7gCH/zZ3WqTkz4FkKiY11hfJ9InX c1PrAP6FrOUWAVZQ31Yef0aD55YQeBwY0/x81Dh9ubVeFuwpzfXSk5D/klB5AByj/oFP vHpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=bBgNab7REdVl+SviiK+uiL0gPCzgPViae4/JyPmHOt8=; b=MyOAhG4UcbRCRMCY10FrjfSdsymH+MTH0tmQfyUlJrMaGBgwR7aDm/KPgHLU/loVGI o9tWfsF6IfkmQSmmNTAMByCofpJYKtqdtmyAu6opkm6lxpL0JhFALTJX6IZiosaQ+6i0 4wWxu8IVN7VRynfz28GBARY5m9uIJkRkt8Uz0QtXcLZKME+NgHfECtWagkB0lE/RBzMe FKHZK5ZOntwy7zh0vBvNGQt367RnfQ+b34LUNC/o0oX/Qjvap8dzCMn7D61nDYElx05b 4p37y/f3NMtV5BpYYtizDoV5Hz3Mh9Rot9XI5W20GmD6Jq4cPzOswxaV4AobCA0Ckxhn IV7A== X-Gm-Message-State: AOAM532H8LF6yS/Fdb8RTCGT1/mBlm4bW1/QXSn0wUxbhmEYopyNkXF0 eDb18J44Y1XKlKdBtETPsASJVSXuAmXIn6snf7LS93BbV4lVlA== X-Google-Smtp-Source: ABdhPJwomSvzotj0SZwAp/e5U23+p3yr4K80gheLIa7c99tpEJD6Yg/4sPwoXPhkhhChR26Ck8TR830HulH4OSaqPSM= X-Received: by 2002:a05:6000:18a7:: with SMTP id b7mr27472554wri.308.1636206062949; Sat, 06 Nov 2021 06:41:02 -0700 (PDT) MIME-Version: 1.0 From: Zach Riggle Date: Sat, 6 Nov 2021 08:40:51 -0500 Message-ID: Subject: Efficient way to map a list of values to multiple processes, then accumulate their output To: Zsh Users Content-Type: multipart/alternative; boundary="000000000000295af705d01ee90c" X-Seq: 27310 Archived-At: X-Loop: zsh-users@zsh.org Errors-To: zsh-users-owner@zsh.org Precedence: list Precedence: bulk Sender: zsh-users-request@zsh.org X-no-archive: yes List-Id: List-Help: List-Subscribe: List-Unsubscribe: List-Post: List-Owner: List-Archive: --000000000000295af705d01ee90c Content-Type: text/plain; charset="UTF-8" Hello all! Thanks in advance for the assistance. Ultimately, I'd like to have some quasi-implementation of xargs -P in pure zsh -- which maps a __function__ to a list of arguments, and returns the results in an array or associative array. xargs cannot be used to run shell functions, so this is out. I think this is a situation that Zsh is not well-suited for, but let's say I have a list of files that I want to perform some processing on, then gather the results when the processing has completed. Simply backgrounding each process with & and then wait'ing on all jobs to complete should work, in theory. However, the output from each invocation may be intermixed, even in the simple case of "one line of output per file" the order of completion is non-deterministic and the output may become garbled. This seems something that would fit nicely with co-processes, but it appears each zsh instance can only have one. Another idea is to farm the output of each to a temporary file, and have an associative array of e.g. [input]=$(mktemp) as well as [input]=job_id and then grabbing the output as each job completes. I expect I am not the first person to try this (equivalent to the Python multiprocessing.map) but I figured it was worth asking! *Zach Riggle* --000000000000295af705d01ee90c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello all!=C2=A0 Thanks in advance for the assistance.
Ultimately, I'd like to have some quasi-implementation = of xargs -P in pure zsh -- which maps a __function__ to a list of arguments= , and returns the results in an array or associative array.=C2=A0 xargs can= not be used to run shell functions, so this is out.

I think this is a situation that Zsh is not well-suited for, but let'= s say I have a list of files that I want to perform some processing on, the= n gather the results when the processing has completed.

Simply backgrounding each process with & and then wait'ing=C2= =A0on all jobs to complete should work, in theory.=C2=A0 However, the outpu= t from each invocation may be intermixed, even in the simple case of "= one line of output per file" the order of completion is non-determinis= tic and the output may become garbled.

This seems = something that would fit nicely with co-processes, but it appears each zsh = instance can only have one.

Another idea is to far= m the output of each to a temporary file, and have an associative array of = e.g. [input]=3D$(mktemp) as well as [input]=3Djob_id and then grabbing the = output as each job completes.

I expect I am not th= e first person to try this (equivalent to the Python multiprocessing.map) b= ut I figured it was worth asking!

Zach Riggle<= /b>
--000000000000295af705d01ee90c--