* wait for the next process to finish
@ 2011-12-12 15:46 Anthony R Fletcher
2011-12-12 16:37 ` TJ Luoma
` (3 more replies)
0 siblings, 4 replies; 19+ messages in thread
From: Anthony R Fletcher @ 2011-12-12 15:46 UTC (permalink / raw)
To: zsh-users
I just realised that the 'wait' command will either wait for specified
jobs or all jobs and nothing in between. The manual says "If job is not
given then all currently active child processes are waited for.".
So
sleep 30 &
sleep 10 &
sleep 30 &
sleep 30 &
wait
waits for all the sleeps to finish.
How can I wait for just the next job to finish?
Anthony.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-12 15:46 wait for the next process to finish Anthony R Fletcher
@ 2011-12-12 16:37 ` TJ Luoma
2011-12-12 19:41 ` Wayne Davison
` (2 subsequent siblings)
3 siblings, 0 replies; 19+ messages in thread
From: TJ Luoma @ 2011-12-12 16:37 UTC (permalink / raw)
To: zsh-users
[-- Attachment #1: Type: text/plain, Size: 714 bytes --]
On Mon, Dec 12, 2011 at 10:46 AM, Anthony R Fletcher <arif@mail.nih.gov>wrote:
> I just realised that the 'wait' command will either wait for specified
> jobs or all jobs and nothing in between. The manual says "If job is not
> given then all currently active child processes are waited for.".
>
> So
> sleep 30 &
> sleep 10 &
> sleep 30 &
> sleep 30 &
> wait
> waits for all the sleeps to finish.
>
> How can I wait for just the next job to finish?
>
Just don't run it in the background.
or do it like this:
(job1 ; job2)
or, if you want to make sure that job2 only runs if job1 succeeds:
(job1 && job2)
TjL
ps - my apologies if I am misunderstanding the question
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-12 15:46 wait for the next process to finish Anthony R Fletcher
2011-12-12 16:37 ` TJ Luoma
@ 2011-12-12 19:41 ` Wayne Davison
2011-12-13 7:15 ` Martin Richter
2011-12-13 16:07 ` Rory Mulvaney
2011-12-13 10:01 ` Peter Stephenson
2011-12-13 17:45 ` Stephane Chazelas
3 siblings, 2 replies; 19+ messages in thread
From: Wayne Davison @ 2011-12-12 19:41 UTC (permalink / raw)
To: zsh-users
[-- Attachment #1: Type: text/plain, Size: 765 bytes --]
On Mon, Dec 12, 2011 at 7:46 AM, Anthony R Fletcher <arif@mail.nih.gov>wrote:
> How can I wait for just the next job to finish?
>
One thing that may help you is TRAPCHLD. Sadly, the signal handler doesn't
tell you what pid it is reacting to, nor the exit code.
TRAPCHLD() {
echo here
oldpids=($pids)
pids=( )
for p in $oldpids; do
if kill -0 $p 2>/dev/null; then
pids+=$p
else
#wait $p # Sadly, this doesn't work
echo $p exited
fi
done
}
pids=( )
sleep 10 &
pids+=$!
sleep 20 &
pids+=$!
(sleep 15; false) &
pids+=$!
echo $pids
wait
echo done
It might be nice to set an environment parameter with the pid and status
info right before the dotrap(SIGCHLD) call in jobs.c.
..wayne..
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-12 19:41 ` Wayne Davison
@ 2011-12-13 7:15 ` Martin Richter
2011-12-13 16:07 ` Rory Mulvaney
1 sibling, 0 replies; 19+ messages in thread
From: Martin Richter @ 2011-12-13 7:15 UTC (permalink / raw)
To: zsh-users
On Monday December 12 2011, Wayne Davison wrote:
> On Mon, Dec 12, 2011 at 7:46 AM, Anthony R Fletcher <arif@mail.nih.gov>wrote:
>
> > How can I wait for just the next job to finish?
This is probably not exactly what you want and it isn't pretty either. But you could use `make' for this and use the -j flag for telling it to process stuff in parallel:
#---------------------------
make -j 4 -f =( cat <<EOF
default: task1 task2 task3 task4 task_finally
task1:
@echo task1
sleep 2
@echo task1 finished
task2:
@echo task2
sleep 7
@echo task2 finished
task3:
@echo task3
sleep 4
@echo task3 finished
task4:
@echo task4
sleep 10
@echo task4 finished
task_finally:
@echo finally
EOF)
#---------------------------
However it is not really customizable and I think it is not really meant to be used that way. If this solution does not already make you shudder, you might also be able to construct something likewise using xargs and the --max-procs switch.
Martin
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-12 15:46 wait for the next process to finish Anthony R Fletcher
2011-12-12 16:37 ` TJ Luoma
2011-12-12 19:41 ` Wayne Davison
@ 2011-12-13 10:01 ` Peter Stephenson
2011-12-13 17:10 ` Bart Schaefer
2011-12-13 17:45 ` Stephane Chazelas
3 siblings, 1 reply; 19+ messages in thread
From: Peter Stephenson @ 2011-12-13 10:01 UTC (permalink / raw)
To: zsh-users
On Mon, 12 Dec 2011 10:46:01 -0500
Anthony R Fletcher <arif@mail.nih.gov> wrote:
> I just realised that the 'wait' command will either wait for specified
> jobs or all jobs and nothing in between. The manual says "If job is not
> given then all currently active child processes are waited for.".
>
> So
> sleep 30 &
> sleep 10 &
> sleep 30 &
> sleep 30 &
> wait
> waits for all the sleeps to finish.
>
> How can I wait for just the next job to finish?
Certainly the shell internals don't help you here. There's code to look
at a specific job, decide if it's still going, and exit when it isn't,
which is behind the wait builtin with an argument. There's nothing to
loop over all jobs, decide what's still going, wait for something to
happen, then work out what it was and hence if it can stop waiting.
The reason waiting for all jobs works is simply that it waits for jobs
in order; if an earlier job exits first it gets handled by the signal
handler, but the shell foreground doesn't notice until the job it's
actually waiting for exits, then it carries on down the list and picks
up anything that's already exited.
--
Peter Stephenson <pws@csr.com> Software Engineer
Tel: +44 (0)1223 692070 Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK
Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-12 19:41 ` Wayne Davison
2011-12-13 7:15 ` Martin Richter
@ 2011-12-13 16:07 ` Rory Mulvaney
2011-12-13 16:45 ` Jérémie Roquet
` (2 more replies)
1 sibling, 3 replies; 19+ messages in thread
From: Rory Mulvaney @ 2011-12-13 16:07 UTC (permalink / raw)
To: zsh-users; +Cc: Wayne Davison, Anthony R Fletcher
On Mon, 12 Dec 2011, Wayne Davison wrote:
> On Mon, Dec 12, 2011 at 7:46 AM, Anthony R Fletcher <arif@mail.nih.gov>wrote:
>
> > How can I wait for just the next job to finish?
> >
>
> One thing that may help you is TRAPCHLD. Sadly, the signal handler doesn't
> tell you what pid it is reacting to, nor the exit code.
>
> TRAPCHLD() {
> echo here
> oldpids=($pids)
> pids=( )
> for p in $oldpids; do
> if kill -0 $p 2>/dev/null; then
> pids+=$p
> else
> #wait $p # Sadly, this doesn't work
> echo $p exited
> fi
> done
> }
> pids=( )
> sleep 10 &
> pids+=$!
> sleep 20 &
> pids+=$!
> (sleep 15; false) &
> pids+=$!
> echo $pids
> wait
> echo done
>
> It might be nice to set an environment parameter with the pid and status
> info right before the dotrap(SIGCHLD) call in jobs.c.
>
> ..wayne..
>
To clarify (I think this is fairly simple), you can supply the process id
as a parameter to 'wait', and though the $! method seems rather clumsy to
retrieve the pid (since you have to retrieve it somehow in the next
command after spawning the background process), it seems to work mostly in
general.
So you could do:
sleep 20000 &
sleep 20 &
pid=$!
wait $pid
That will just wait for the sleep 20 process to complete while the sleep
20000 process still runs in the background.
For further reference, I see that $! is documented in sec. 15.5
(Parameters set by the shell) in the zsh texinfo manual.
-Rory
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-13 16:07 ` Rory Mulvaney
@ 2011-12-13 16:45 ` Jérémie Roquet
2011-12-13 16:59 ` Anthony R Fletcher
2011-12-13 16:49 ` Christoph (Stucki) von Stuckrad
2011-12-13 17:04 ` Rory Mulvaney
2 siblings, 1 reply; 19+ messages in thread
From: Jérémie Roquet @ 2011-12-13 16:45 UTC (permalink / raw)
To: Rory Mulvaney; +Cc: zsh-users, Wayne Davison, Anthony R Fletcher
Hi,
2011/12/13 Rory Mulvaney <rorymulv@gmail.com>:
> To clarify (I think this is fairly simple), you can supply the process id
> as a parameter to 'wait', and though the $! method seems rather clumsy to
> retrieve the pid (since you have to retrieve it somehow in the next
> command after spawning the background process), it seems to work mostly in
> general.
>
> So you could do:
>
> sleep 20000 &
> sleep 20 &
> pid=$!
> wait $pid
>
> That will just wait for the sleep 20 process to complete while the sleep
> 20000 process still runs in the background.
Actually, it'll always wait for the last spawned job, not for the
first job to finish.
If you spawn them in the reverse order, ie:
sleep 20 &
sleep 20000 &
pid=$!
wait $pid
This will wait for the sleep 20000 process, even if the sleep 20 has
finished for long.
Best regards,
--
Jérémie
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-13 16:07 ` Rory Mulvaney
2011-12-13 16:45 ` Jérémie Roquet
@ 2011-12-13 16:49 ` Christoph (Stucki) von Stuckrad
2011-12-13 17:31 ` Bart Schaefer
2011-12-13 17:04 ` Rory Mulvaney
2 siblings, 1 reply; 19+ messages in thread
From: Christoph (Stucki) von Stuckrad @ 2011-12-13 16:49 UTC (permalink / raw)
To: zsh-users
On Tue, 13 Dec 2011, Rory Mulvaney wrote:
> > On Mon, Dec 12, 2011 at 7:46 AM, Anthony R Fletcher <arif@mail.nih.gov>wrote:
> >
> > > How can I wait for just the next job to finish?
> > >
> >
> > One thing that may help you is TRAPCHLD. Sadly, the signal handler doesn't
> > tell you what pid it is reacting to, nor the exit code.
> >
> > TRAPCHLD() {
...
> > }
...
> > It might be nice to set an environment parameter with the pid and status
> > info right before the dotrap(SIGCHLD) call in jobs.c.
...
This sounds very promising, because you then could 'parallelize'
a loop by feeding it new processes when 'one of the runnig' dies.
I hope for something in the line of:
TRAPCHILD() { ... ... ... }
pids=( )
for i in some paramters given
do
if [[ $#pids >= $MAXPARALLEL ]]
then sleep 1 # and TRAPCHILD will remove 'done's
else
call $i &
pids+=$!
fi
done
SO one might be able to keep a bunch of CPUS working
for 'one given loop' without needing to know which
case is done first? (And only if the parallelities
do not break something like tempfiles, stderr, etc ...)
Stucki
--
Christoph von Stuckrad * * |nickname |Mail <stucki@mi.fu-berlin.de> \
Freie Universitaet Berlin |/_*|'stucki' |Tel(Mo.,Mi.):+49 30 838-75 459|
Mathematik & Informatik EDV |\ *|if online| (Di,Do,Fr):+49 30 77 39 6600|
Takustr. 9 / 14195 Berlin * * |on IRCnet|Fax(home): +49 30 77 39 6601/
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-13 16:45 ` Jérémie Roquet
@ 2011-12-13 16:59 ` Anthony R Fletcher
2011-12-13 18:08 ` Daniel Shahaf
0 siblings, 1 reply; 19+ messages in thread
From: Anthony R Fletcher @ 2011-12-13 16:59 UTC (permalink / raw)
To: zsh-users
> 2011/12/13 Rory Mulvaney <rorymulv@gmail.com>:
> > To clarify (I think this is fairly simple), you can supply the process id
> > as a parameter to 'wait', and though the $! method seems rather clumsy to
> > retrieve the pid (since you have to retrieve it somehow in the next
> > command after spawning the background process), it seems to work mostly in
> > general.
> >
> > So you could do:
> >
> > sleep 20000 &
> > sleep 20 &
> > pid=$!
> > wait $pid
> >
> > That will just wait for the sleep 20 process to complete while the sleep
> > 20000 process still runs in the background.
>
> Actually, it'll always wait for the last spawned job, not for the
> first job to finish.
>
> If you spawn them in the reverse order, ie:
>
> sleep 20 &
> sleep 20000 &
> pid=$!
> wait $pid
>
> This will wait for the sleep 20000 process, even if the sleep 20 has
> finished for long.
Yes, the need to to wait for any of the processes to end so I can start
another. Basically a poor man's batch system.
Anthony.
--
Anthony R Fletcher
Room 2033, Building 12A, http://dcb.cit.nih.gov/~arif
National Institutes of Health, arif@mail.nih.gov
12A South Drive, Bethesda, Phone: (+1) 301 402 1741.
MD 20892-5624, USA.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-13 16:07 ` Rory Mulvaney
2011-12-13 16:45 ` Jérémie Roquet
2011-12-13 16:49 ` Christoph (Stucki) von Stuckrad
@ 2011-12-13 17:04 ` Rory Mulvaney
2011-12-13 17:32 ` Anthony R Fletcher
2 siblings, 1 reply; 19+ messages in thread
From: Rory Mulvaney @ 2011-12-13 17:04 UTC (permalink / raw)
To: Rory Mulvaney; +Cc: zsh-users, Wayne Davison, Anthony R Fletcher
On Tue, 13 Dec 2011, Rory Mulvaney wrote:
> On Mon, 12 Dec 2011, Wayne Davison wrote:
>
> > On Mon, Dec 12, 2011 at 7:46 AM, Anthony R Fletcher <arif@mail.nih.gov>wrote:
> >
> > > How can I wait for just the next job to finish?
> > >
> >
> > One thing that may help you is TRAPCHLD. Sadly, the signal handler doesn't
> > tell you what pid it is reacting to, nor the exit code.
> >
> > TRAPCHLD() {
> > echo here
> > oldpids=($pids)
> > pids=( )
> > for p in $oldpids; do
> > if kill -0 $p 2>/dev/null; then
> > pids+=$p
> > else
> > #wait $p # Sadly, this doesn't work
> > echo $p exited
> > fi
> > done
> > }
> > pids=( )
> > sleep 10 &
> > pids+=$!
> > sleep 20 &
> > pids+=$!
> > (sleep 15; false) &
> > pids+=$!
> > echo $pids
> > wait
> > echo done
> >
> > It might be nice to set an environment parameter with the pid and status
> > info right before the dotrap(SIGCHLD) call in jobs.c.
> >
> > ..wayne..
> >
>
> To clarify (I think this is fairly simple), you can supply the process id
> as a parameter to 'wait', and though the $! method seems rather clumsy to
> retrieve the pid (since you have to retrieve it somehow in the next
> command after spawning the background process), it seems to work mostly in
> general.
>
> So you could do:
>
> sleep 20000 &
> sleep 20 &
> pid=$!
> wait $pid
>
> That will just wait for the sleep 20 process to complete while the sleep
> 20000 process still runs in the background.
>
> For further reference, I see that $! is documented in sec. 15.5
> (Parameters set by the shell) in the zsh texinfo manual.
>
Sorry, now I realized that you want to want to wait for whichever
background process finishes first. I know that select (zsh/zselect) is
used for this type of thing, which blocks until some file descriptor is
ready for reading or writing. It would possible but sort of complicated
to hack that functionality into the case of waiting for pids. I can
imagine using coproc to get a file descriptor for each of the background
processes.
Something like this (requiring a zsh version with anonymous functions):
pids=( )
R=( $RANDOM $RANDOM )
echo $R
coproc () { sleep $(( $R[1]/32767.0*20 )) ; echo $$ }
pids+=$!
# copy the coproc fd
exec 3<&p
coproc () { sleep $(( $R[2]/32767.0*20 )) ; echo $$ }
pids+=$!
exec 4<&p
zmodload zsh/zselect
zselect -r 3 4
echo $reply
Then one could parse $reply.
I don't know, maybe there's a much easier way to create file descriptors
for those processes?
-Rory
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-13 10:01 ` Peter Stephenson
@ 2011-12-13 17:10 ` Bart Schaefer
2011-12-13 20:42 ` Rory Mulvaney
0 siblings, 1 reply; 19+ messages in thread
From: Bart Schaefer @ 2011-12-13 17:10 UTC (permalink / raw)
To: zsh-users
On Dec 13, 10:01am, Peter Stephenson wrote:
} Subject: Re: wait for the next process to finish
}
} On Mon, 12 Dec 2011 10:46:01 -0500
} Anthony R Fletcher <arif@mail.nih.gov> wrote:
} > I just realised that the 'wait' command will either wait for specified
} > jobs or all jobs and nothing in between. The manual says "If job is not
} > given then all currently active child processes are waited for.".
} >
} > So
} > sleep 30 &
} > sleep 10 &
} > sleep 30 &
} > sleep 30 &
} > wait
} > waits for all the sleeps to finish.
} >
} > How can I wait for just the next job to finish?
}
} Certainly the shell internals don't help you here. There's code to look
} at a specific job, decide if it's still going, and exit when it isn't,
} which is behind the wait builtin with an argument. There's nothing to
} loop over all jobs, decide what's still going, wait for something to
} happen, then work out what it was and hence if it can stop waiting.
What's "just the next job" mean here?
Wait until any one of the four jobs finishes, or wait until a specific
job (e.g., the first one started, and then the second one, and then
the third, etc.) finishes?
I think there have been some answers on this thread that assume one
meaning and some the other, and therefore are contradictory. If it's
the first meaning (any job) then Wayne's TRAPCHLD suggestion is the
most helpful, but it's true the trap doesn't get useful information
about which child set it off, and on top of that the handler is not
called until the child is already removed from the job table, so even
if you could figure out which one it was you can't "wait" to get its
exit status (I don't think).
In the realm of ugly hacks, you could run each child as a coprocess
that ends with "print $?". Stash away the coprocess file descriptor
each time you start one, and then when they're all running, start a
loop polling that list of descriptors with "read -t" until you find
one from which you can read. What you read will be the exit status.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-13 16:49 ` Christoph (Stucki) von Stuckrad
@ 2011-12-13 17:31 ` Bart Schaefer
0 siblings, 0 replies; 19+ messages in thread
From: Bart Schaefer @ 2011-12-13 17:31 UTC (permalink / raw)
To: zsh-users
On Dec 13, 5:49pm, Christoph (Stucki) von Stuckrad wrote:
}
} This sounds very promising, because you then could 'parallelize'
} a loop by feeding it new processes when 'one of the runnig' dies.
That much you can do already.
TRAPCHLD() {
while (( ${#jobstates} < MAXPARALLEL ))
do the background task &
done
}
(Of course you need some sort of other loop-ending condition, but you
get the idea.)
You don't really even need the trap for this, your original loop will
work fine as long as you examine ${(k)jobstates} as the list of PIDs
instead of trying to maintain your own array.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-13 17:04 ` Rory Mulvaney
@ 2011-12-13 17:32 ` Anthony R Fletcher
2011-12-13 18:04 ` Jérémie Roquet
0 siblings, 1 reply; 19+ messages in thread
From: Anthony R Fletcher @ 2011-12-13 17:32 UTC (permalink / raw)
To: zsh-users
On 13 Dec 2011 at 11:04:44, Rory Mulvaney wrote:
> pids=( )
> R=( $RANDOM $RANDOM )
> echo $R
> coproc () { sleep $(( $R[1]/32767.0*20 )) ; echo $$ }
> pids+=$!
> # copy the coproc fd
> exec 3<&p
> coproc () { sleep $(( $R[2]/32767.0*20 )) ; echo $$ }
> pids+=$!
> exec 4<&p
> zmodload zsh/zselect
> zselect -r 3 4
> echo $reply
>
> Then one could parse $reply.
>
> I don't know, maybe there's a much easier way to create file descriptors
> for those processes?
>
> -Rory
I was using this loop
for f in *.txt
do
what-ever-has-to-happen $f &
pwait 20
done
where pwait is a function
function pwait() {
while [ $(jobs -rp | wc -l) -ge $1 ]
do
# sleep 10
wait
done
}
(see http://superuser.com/questions/158165/parallel-shell-loops for one
example amongst many).
I was hoping that using "wait" would be better than using a "sleep 10".
But wait waits for them all and not just he last to finish. The nice
thing is that this is a small, easy loop and all the brains is in the
barrier function pwait. Sadly a sleep for 10 seconds is completely
arbitrary.
Anthony.
--
Anthony R Fletcher
Room 2033, Building 12A, http://dcb.cit.nih.gov/~arif
National Institutes of Health, arif@mail.nih.gov
12A South Drive, Bethesda, Phone: (+1) 301 402 1741.
MD 20892-5624, USA.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-12 15:46 wait for the next process to finish Anthony R Fletcher
` (2 preceding siblings ...)
2011-12-13 10:01 ` Peter Stephenson
@ 2011-12-13 17:45 ` Stephane Chazelas
3 siblings, 0 replies; 19+ messages in thread
From: Stephane Chazelas @ 2011-12-13 17:45 UTC (permalink / raw)
To: zsh-users
2011-12-12 10:46:01 -0500, Anthony R Fletcher:
> I just realised that the 'wait' command will either wait for specified
> jobs or all jobs and nothing in between. The manual says "If job is not
> given then all currently active child processes are waited for.".
>
> So
> sleep 30 &
> sleep 10 &
> sleep 30 &
> sleep 30 &
> wait
> waits for all the sleeps to finish.
>
> How can I wait for just the next job to finish?
[...]
What about:
{
(sleep 4; echo A) &
(sleep 1; echo B) &
(sleep 8; echo C) &
(sleep 2; echo D) &
(sleep 7; echo E) &
(sleep 3; echo F) &
} | while read task; do
date +"%T: $task finished"
done
17:44:17: B finished
17:44:18: D finished
17:44:19: F finished
17:44:20: A finished
17:44:23: E finished
17:44:24: C finished
--
Stephane
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-13 17:32 ` Anthony R Fletcher
@ 2011-12-13 18:04 ` Jérémie Roquet
2011-12-13 18:20 ` Stephane Chazelas
0 siblings, 1 reply; 19+ messages in thread
From: Jérémie Roquet @ 2011-12-13 18:04 UTC (permalink / raw)
To: zsh-users
2011/12/13 Anthony R Fletcher <arif@mail.nih.gov>:
> I was using this loop
>
> for f in *.txt
> do
> what-ever-has-to-happen $f &
> pwait 20
> done
>
> where pwait is a function
>
> function pwait() {
> while [ $(jobs -rp | wc -l) -ge $1 ]
> do
> # sleep 10
> wait
> done
> }
>
> (see http://superuser.com/questions/158165/parallel-shell-loops for one
> example amongst many).
>
> I was hoping that using "wait" would be better than using a "sleep 10".
> But wait waits for them all and not just he last to finish. The nice
> thing is that this is a small, easy loop and all the brains is in the
> barrier function pwait. Sadly a sleep for 10 seconds is completely
> arbitrary.
I'm wondering if you're not just looking for zargs :-)
autoload -U zargs
zargs -n 1 -P 20 *.txt -- what-ever-has-to-happen
Best regards,
--
Jérémie
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-13 16:59 ` Anthony R Fletcher
@ 2011-12-13 18:08 ` Daniel Shahaf
0 siblings, 0 replies; 19+ messages in thread
From: Daniel Shahaf @ 2011-12-13 18:08 UTC (permalink / raw)
To: zsh-users
Anthony R Fletcher wrote on Tue, Dec 13, 2011 at 11:59:29 -0500:
> > 2011/12/13 Rory Mulvaney <rorymulv@gmail.com>:
> > > To clarify (I think this is fairly simple), you can supply the process id
> > > as a parameter to 'wait', and though the $! method seems rather clumsy to
> > > retrieve the pid (since you have to retrieve it somehow in the next
> > > command after spawning the background process), it seems to work mostly in
> > > general.
> > >
> > > So you could do:
> > >
> > > sleep 20000 &
> > > sleep 20 &
> > > pid=$!
> > > wait $pid
> > >
> > > That will just wait for the sleep 20 process to complete while the sleep
> > > 20000 process still runs in the background.
> >
> > Actually, it'll always wait for the last spawned job, not for the
> > first job to finish.
> >
> > If you spawn them in the reverse order, ie:
> >
> > sleep 20 &
> > sleep 20000 &
> > pid=$!
> > wait $pid
> >
> > This will wait for the sleep 20000 process, even if the sleep 20 has
> > finished for long.
>
> Yes, the need to to wait for any of the processes to end so I can start
> another. Basically a poor man's batch system.
'xargs -P' and 'make -j' were already suggested upthread; is the task
you're trying to parallelize not suitable for those approaches?
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-13 18:04 ` Jérémie Roquet
@ 2011-12-13 18:20 ` Stephane Chazelas
2011-12-13 19:19 ` Anthony R Fletcher
0 siblings, 1 reply; 19+ messages in thread
From: Stephane Chazelas @ 2011-12-13 18:20 UTC (permalink / raw)
To: Jérémie Roquet; +Cc: zsh-users
2011-12-13 19:04:12 +0100, Jérémie Roquet:
[...]
> I'm wondering if you're not just looking for zargs :-)
[...]
Or GNU parallel or pexec.
--
Stephane
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-13 18:20 ` Stephane Chazelas
@ 2011-12-13 19:19 ` Anthony R Fletcher
0 siblings, 0 replies; 19+ messages in thread
From: Anthony R Fletcher @ 2011-12-13 19:19 UTC (permalink / raw)
To: Jérémie Roquet, zsh-users
On 13 Dec 2011 at 18:20:03, Stephane Chazelas wrote:
> 2011-12-13 19:04:12 +0100, Jérémie Roquet:
> [...]
> > I'm wondering if you're not just looking for zargs :-)
> [...]
>
> Or GNU parallel or pexec.
>
I considered GNU parallel but the loop seems do much easier for what I
wanted to do. pexec needs to be compiled - too hard for this job. The
loop idea was just too easy to use.
I didn't know about zargs but I'll play with it. Thanks.
Anthony.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: wait for the next process to finish
2011-12-13 17:10 ` Bart Schaefer
@ 2011-12-13 20:42 ` Rory Mulvaney
0 siblings, 0 replies; 19+ messages in thread
From: Rory Mulvaney @ 2011-12-13 20:42 UTC (permalink / raw)
To: Bart Schaefer; +Cc: zsh-users, rorymulv
On Tue, 13 Dec 2011, Bart Schaefer wrote:
>
> In the realm of ugly hacks, you could run each child as a coprocess
> that ends with "print $?". Stash away the coprocess file descriptor
> each time you start one, and then when they're all running, start a
> loop polling that list of descriptors with "read -t" until you find
> one from which you can read. What you read will be the exit status.
>
Yes, this is what I demonstrated in my second email, but the idea about
using TRAPCHLD is probably easiest, since you could have your child
processes print out any info about the process such as the return code,
redirecting it to a single file descriptor owned by a designated
information-relay coproc (which merely echo's its stdin to stdout), and
then parse the output from the coproc in TRAPCHLD, which is apparently one
of the available trap functions running whenever a signal arrives, since
SIGCHLD is a signal from signal(7) (I didn't realize that before).
On Tue, 13 Dec 2011, Bart Schaefer wrote:
>
> TRAPCHLD() {
> while (( ${#jobstates} < MAXPARALLEL ))
> do the background task &
> done
> }
>
> (Of course you need some sort of other loop-ending condition, but you
> get the idea.)
>
> You don't really even need the trap for this, your original loop will
> work fine as long as you examine ${(k)jobstates} as the list of PIDs
> instead of trying to maintain your own array.
>
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2011-12-13 20:45 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-12-12 15:46 wait for the next process to finish Anthony R Fletcher
2011-12-12 16:37 ` TJ Luoma
2011-12-12 19:41 ` Wayne Davison
2011-12-13 7:15 ` Martin Richter
2011-12-13 16:07 ` Rory Mulvaney
2011-12-13 16:45 ` Jérémie Roquet
2011-12-13 16:59 ` Anthony R Fletcher
2011-12-13 18:08 ` Daniel Shahaf
2011-12-13 16:49 ` Christoph (Stucki) von Stuckrad
2011-12-13 17:31 ` Bart Schaefer
2011-12-13 17:04 ` Rory Mulvaney
2011-12-13 17:32 ` Anthony R Fletcher
2011-12-13 18:04 ` Jérémie Roquet
2011-12-13 18:20 ` Stephane Chazelas
2011-12-13 19:19 ` Anthony R Fletcher
2011-12-13 10:01 ` Peter Stephenson
2011-12-13 17:10 ` Bart Schaefer
2011-12-13 20:42 ` Rory Mulvaney
2011-12-13 17:45 ` Stephane Chazelas
Code repositories for project(s) associated with this public inbox
https://git.vuxu.org/mirror/zsh/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).