help / color / mirror / code / Atom feed
From: Perry Smith <pedz@easesoftware.com>
To: zsh-users@zsh.org
Subject: Re: Parallel processing
Date: Mon, 28 Mar 2022 08:31:57 -0500	[thread overview]
Message-ID: <C05C228C-07DE-4B43-BE27-0FABD5B6258C@easesoftware.com> (raw)
In-Reply-To: <1E0E1226-E3E8-40AD-87CD-93A602B1B08B@easesoftware.com>

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

Perhaps I need to think about this another way and that is via “locking”.

From a 10,000 foot view, I could map each target to a file (path) and do the old fashion practice of putting the PID in the file and do it atomically and with the ability to detect that the owner of the lock might have died, etc etc etc.

Now just spin off N tasks each with a complete list of targets.  The basic flow would be for each item in the list of targets, see if it is done.  If it isn’t done, see if it is locked.  If it isn’t locked, set the lock (atomically) and then do the processing to create the target.

Now, if I want 4 processes running, I can just start four jobs in the back ground.

> On Mar 24, 2022, at 23:34, Perry Smith <pedz@easesoftware.com> wrote:
> Has something like prll (parallel) https://github.com/exzombie/prll <https://github.com/exzombie/prll> been added to zsh?
> I need to do about 20 commands.  Each will take several hours to perhaps days.  I’d like to start some fixed number like 4 jobs that are running.  The others are waiting for one of the others to get finished.
> I tried creating a Makefile and use the -j option in make but my targets have spaces in them and make doesn’t like that.  (This is on a BSD system.)
> Thank you,
> Perry

[-- Attachment #2: Type: text/html, Size: 2269 bytes --]

  parent reply	other threads:[~2022-03-28 13:33 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-03-25  4:34 Perry Smith
2022-03-25 18:27 ` Bart Schaefer
2022-03-26 18:10   ` Philippe Troin
2022-03-26 22:19     ` Bart Schaefer
2022-03-27 17:32       ` Philippe Troin
2022-03-27 17:42         ` Bart Schaefer
2022-03-27 19:23 ` Dominik Vogt
2022-03-28 13:26   ` Perry Smith
2022-03-28 13:31 ` Perry Smith [this message]
2022-03-26  4:34 jdh
2022-03-27 18:37 jdh
2022-03-27 19:06 ` Bart Schaefer
2022-03-27 19:24   ` Ray Andrews
2022-03-28 13:47 ` Perry Smith

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=C05C228C-07DE-4B43-BE27-0FABD5B6258C@easesoftware.com \
    --to=pedz@easesoftware.com \
    --cc=zsh-users@zsh.org \
    --subject='Re: Parallel processing' \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).