Re: Parallel processing

[Perry Smith <pedz@easesoftware.com>]

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

Perhaps I need to think about this another way and that is via “locking”.

From a 10,000 foot view, I could map each target to a file (path) and do the old fashion practice of putting the PID in the file and do it atomically and with the ability to detect that the owner of the lock might have died, etc etc etc.

Now just spin off N tasks each with a complete list of targets.  The basic flow would be for each item in the list of targets, see if it is done.  If it isn’t done, see if it is locked.  If it isn’t locked, set the lock (atomically) and then do the processing to create the target.

Now, if I want 4 processes running, I can just start four jobs in the back ground.

> On Mar 24, 2022, at 23:34, Perry Smith <pedz@easesoftware.com> wrote:
> 
> Has something like prll (parallel) https://github.com/exzombie/prll <https://github.com/exzombie/prll> been added to zsh?
> 
> I need to do about 20 commands.  Each will take several hours to perhaps days.  I’d like to start some fixed number like 4 jobs that are running.  The others are waiting for one of the others to get finished.
> 
> I tried creating a Makefile and use the -j option in make but my targets have spaces in them and make doesn’t like that.  (This is on a BSD system.)
> 
> Thank you,
> Perry


[-- Attachment #2: Type: text/html, Size: 2269 bytes --]