Running N jobs from M all the time

Running N jobs from M all the time

[Szabolcs Szakacsits]

Hi,

I'm a new subscriber but using this lovely, powerful - well, as I
experienced far the best - shell for years. I can't live without it -
thank you for zsh!

My question would be there is an easy way for $SUBJECT? It would be
useful for SMP machines (e.g. running parallel exactly 4 graphic
processing utilities on a 4-way Xeon for/out of 1000 images) or when a
singlethreaded networking software can be split up into multiply parts
(e.g. mirroring, rsyncing different hosts) on a high-bandwidth host
thus a slow remote host or networking problem couldn't slow
down/stop the whole process. 

I know this can be done with ps, grep, wait, etc but a bit painful and
not elegant/modern I'd like to use a cleaner way. RTFM is welcomed but
I've read the docs and searched the zsh archivum for similar question
but couldn't find answer.

Thanks,

Szaka, who is disappointed because still none of the Linux
distribution makers select a well configured zsh as the default 
shell.

Re: Running N jobs from M all the time

[Bart Schaefer]

On Oct 12,  6:38am, Szabolcs Szakacsits wrote:
} Subject: Running N jobs from M all the time
}
} My question would be there is an easy way for $SUBJECT?

If I'm understanding you correctly, you want to do something like

repeat 1000 do
    if (( number_of_jobs >= number_of_processors )); then
	wait $any_job
    fi
    command &
done

And the complaint is that "wait" can only wait for one specific job or
for all of them, not for "the next one that exits."

} I know this can be done with ps, grep, wait, etc but a bit painful and
} not elegant/modern I'd like to use a cleaner way.

You've just discovered another use for the coprocess.  The rest of this
is just standard parallel programming stuff.

function run_parallel {
    # Runs N copies of command at once until M copies have run.

    emulate -L zsh
    if (( $# < 3 )); then
	print "usage: run_parallel N M command [args ...]" >&2
	return 1
    fi

    integer N=$1 M=$2
    shift 2

    if ((M <= N)); then
	repeat $M do $==* & done
	wait
    else
	# Set up a loop to read and write one byte at a time.  This
	# semaphores between the parallel children and the parent zsh.  
	# It's important to pass exactly one byte here and not rely on
	# reading complete lines, lest a race condition develop.

	# Unfortunately, a race still can occur, it's just less likely.
	# Maybe SMP systems are better about multiple writers on one FD?

	coproc while ((--M > 0)) && read -u0k; do print -n .; done

	# Start the first N children

	repeat $N do
	    ((--M))
	    { $==* ; print -np . } &
	done

	# Wait for the semaphore from each child and start another as
	# soon as we get it, up to M children.

	while ((--M > 0)) && read -pk; do
	    { $==* ; print -np . } &
	done

	# Wait for the last N semaphores.

	repeat $N do read -pk; done
    fi

    return $?
}

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: Running N jobs from M all the time

[Szabolcs Szakacsits]

On Tue, 12 Oct 1999, Bart Schaefer wrote:
> On Oct 12,  6:38am, Szabolcs Szakacsits wrote:
> } Subject: Running N jobs from M all the time
> }
> } My question would be there is an easy way for $SUBJECT?
> 
> If I'm understanding you correctly, you want to do something like
> 
> repeat 1000 do
>     if (( number_of_jobs >= number_of_processors )); then
> 	wait $any_job
>     fi
>     command &
> done

Yes and thanks for the answer and run_parallel function. In brief
what and how I'd really like to see is e.g.

for i in * PARALLEL N ; do job $i ; done

Would it be very difficult to add these kind of feature? This would
always work when I'll do an

[anyshell] zsh
[zsh :)] 

on any machine and no matter what zsh functions/extensions are
installed/configured.

	Szaka

Re: Running N jobs from M all the time

[Adam Spiers]

Szabolcs Szakacsits (szaka@sienet.hu) wrote:
> Szaka, who is disappointed because still none of the Linux
> distribution makers select a well configured zsh as the default 
> shell.

I share this disappointment (although maybe Mandrake has it as
default?).  My hope is that when more people notice features such as
the new completion system, it will start catching on very quickly, and
eventually become more mainstream.  Roll on 3.2.x ...