wait for the next process to finish

wait for the next process to finish

[Anthony R Fletcher]

I just realised that the 'wait' command will either wait for specified
jobs or all jobs and nothing in between. The manual says "If job is not
given then all currently active child processes are waited for.".

So 
	sleep 30 &
	sleep 10 &
	sleep 30 &
	sleep 30 &
	wait
waits for all the sleeps to finish.

How can I wait for just the next job to finish?

			Anthony.

Re: wait for the next process to finish

[TJ Luoma]

[-- Attachment #1: Type: text/plain, Size: 714 bytes --]

On Mon, Dec 12, 2011 at 10:46 AM, Anthony R Fletcher <arif@mail.nih.gov>wrote:

> I just realised that the 'wait' command will either wait for specified
> jobs or all jobs and nothing in between. The manual says "If job is not
> given then all currently active child processes are waited for.".
>
> So
>        sleep 30 &
>        sleep 10 &
>        sleep 30 &
>        sleep 30 &
>        wait
> waits for all the sleeps to finish.
>
> How can I wait for just the next job to finish?
>

Just don't run it in the background.

or do it like this:

(job1 ; job2)

or, if you want to make sure that job2 only runs if job1 succeeds:

(job1 && job2)

TjL

ps - my apologies if I am misunderstanding the question




>

Re: wait for the next process to finish

[Wayne Davison]

[-- Attachment #1: Type: text/plain, Size: 765 bytes --]

On Mon, Dec 12, 2011 at 7:46 AM, Anthony R Fletcher <arif@mail.nih.gov>wrote:

> How can I wait for just the next job to finish?
>

One thing that may help you is TRAPCHLD.  Sadly, the signal handler doesn't
tell you what pid it is reacting to, nor the exit code.

TRAPCHLD() {
    echo here
    oldpids=($pids)
    pids=( )
    for p in $oldpids; do
        if kill -0 $p 2>/dev/null; then
            pids+=$p
        else
            #wait $p # Sadly, this doesn't work
            echo $p exited
        fi
    done
}
pids=( )
sleep 10 &
pids+=$!
sleep 20 &
pids+=$!
(sleep 15; false) &
pids+=$!
echo $pids
wait
echo done

It might be nice to set an environment parameter with the pid and status
info right before the dotrap(SIGCHLD) call in jobs.c.

..wayne..

Re: wait for the next process to finish

[Martin Richter]

On Monday December 12 2011, Wayne Davison wrote:
> On Mon, Dec 12, 2011 at 7:46 AM, Anthony R Fletcher <arif@mail.nih.gov>wrote:
> 
> > How can I wait for just the next job to finish?

This is probably not exactly what you want and it isn't pretty either. But you could use `make' for this and use the -j flag for telling it to process stuff in parallel:


#---------------------------

make -j 4 -f =( cat <<EOF
default: task1 task2 task3 task4 task_finally

task1:
        @echo task1
        sleep 2
        @echo task1 finished

task2:
        @echo task2
        sleep 7
        @echo task2 finished

task3:
        @echo task3
        sleep 4
        @echo task3 finished

task4:
        @echo task4
        sleep 10
        @echo task4 finished

task_finally:
        @echo finally
EOF)

#---------------------------

However it is not really customizable and I think it is not really meant to be used that way. If this solution does not already make you shudder, you might also be able to construct something likewise using xargs and the --max-procs switch.

Martin

Re: wait for the next process to finish

[Rory Mulvaney]

On Mon, 12 Dec 2011, Wayne Davison wrote:

> On Mon, Dec 12, 2011 at 7:46 AM, Anthony R Fletcher <arif@mail.nih.gov>wrote:
> 
> > How can I wait for just the next job to finish?
> >
> 
> One thing that may help you is TRAPCHLD.  Sadly, the signal handler doesn't
> tell you what pid it is reacting to, nor the exit code.
> 
> TRAPCHLD() {
>     echo here
>     oldpids=($pids)
>     pids=( )
>     for p in $oldpids; do
>         if kill -0 $p 2>/dev/null; then
>             pids+=$p
>         else
>             #wait $p # Sadly, this doesn't work
>             echo $p exited
>         fi
>     done
> }
> pids=( )
> sleep 10 &
> pids+=$!
> sleep 20 &
> pids+=$!
> (sleep 15; false) &
> pids+=$!
> echo $pids
> wait
> echo done
> 
> It might be nice to set an environment parameter with the pid and status
> info right before the dotrap(SIGCHLD) call in jobs.c.
> 
> ..wayne..
> 

To clarify (I think this is fairly simple), you can supply the process id 
as a parameter to 'wait', and though the $! method seems rather clumsy to 
retrieve the pid (since you have to retrieve it somehow in the next 
command after spawning the background process), it seems to work mostly in 
general.

So you could do:

sleep 20000 &
sleep 20 &
pid=$!
wait $pid

That will just wait for the sleep 20 process to complete while the sleep 
20000 process still runs in the background.

For further reference, I see that $! is documented in sec. 15.5 
(Parameters set by the shell) in the zsh texinfo manual.

-Rory

Re: wait for the next process to finish

[Jérémie Roquet]

Hi,

2011/12/13 Rory Mulvaney <rorymulv@gmail.com>:
> To clarify (I think this is fairly simple), you can supply the process id
> as a parameter to 'wait', and though the $! method seems rather clumsy to
> retrieve the pid (since you have to retrieve it somehow in the next
> command after spawning the background process), it seems to work mostly in
> general.
>
> So you could do:
>
> sleep 20000 &
> sleep 20 &
> pid=$!
> wait $pid
>
> That will just wait for the sleep 20 process to complete while the sleep
> 20000 process still runs in the background.

Actually, it'll always wait for the last spawned job, not for the
first job to finish.

If you spawn them in the reverse order, ie:

sleep 20 &
sleep 20000 &
pid=$!
wait $pid

This will wait for the sleep 20000 process, even if the sleep 20 has
finished for long.

Best regards,

-- 
Jérémie

Re: wait for the next process to finish

[Anthony R Fletcher]

> 2011/12/13 Rory Mulvaney <rorymulv@gmail.com>:
> > To clarify (I think this is fairly simple), you can supply the process id
> > as a parameter to 'wait', and though the $! method seems rather clumsy to
> > retrieve the pid (since you have to retrieve it somehow in the next
> > command after spawning the background process), it seems to work mostly in
> > general.
> >
> > So you could do:
> >
> > sleep 20000 &
> > sleep 20 &
> > pid=$!
> > wait $pid
> >
> > That will just wait for the sleep 20 process to complete while the sleep
> > 20000 process still runs in the background.
> 
> Actually, it'll always wait for the last spawned job, not for the
> first job to finish.
> 
> If you spawn them in the reverse order, ie:
> 
> sleep 20 &
> sleep 20000 &
> pid=$!
> wait $pid
> 
> This will wait for the sleep 20000 process, even if the sleep 20 has
> finished for long.

Yes, the need to to wait for any of the processes to end so I can start
another. Basically a poor man's batch system.

			Anthony.

-- 
Anthony R Fletcher        
  Room 2033, Building 12A,        http://dcb.cit.nih.gov/~arif
  National Institutes of Health,  arif@mail.nih.gov
  12A South Drive, Bethesda,      Phone: (+1) 301 402 1741.
  MD 20892-5624, USA.

Re: wait for the next process to finish

[Daniel Shahaf]

Anthony R Fletcher wrote on Tue, Dec 13, 2011 at 11:59:29 -0500:
> > 2011/12/13 Rory Mulvaney <rorymulv@gmail.com>:
> > > To clarify (I think this is fairly simple), you can supply the process id
> > > as a parameter to 'wait', and though the $! method seems rather clumsy to
> > > retrieve the pid (since you have to retrieve it somehow in the next
> > > command after spawning the background process), it seems to work mostly in
> > > general.
> > >
> > > So you could do:
> > >
> > > sleep 20000 &
> > > sleep 20 &
> > > pid=$!
> > > wait $pid
> > >
> > > That will just wait for the sleep 20 process to complete while the sleep
> > > 20000 process still runs in the background.
> > 
> > Actually, it'll always wait for the last spawned job, not for the
> > first job to finish.
> > 
> > If you spawn them in the reverse order, ie:
> > 
> > sleep 20 &
> > sleep 20000 &
> > pid=$!
> > wait $pid
> > 
> > This will wait for the sleep 20000 process, even if the sleep 20 has
> > finished for long.
> 
> Yes, the need to to wait for any of the processes to end so I can start
> another. Basically a poor man's batch system.

'xargs -P' and 'make -j' were already suggested upthread; is the task
you're trying to parallelize not suitable for those approaches?

Re: wait for the next process to finish

[Christoph (Stucki) von Stuckrad]

On Tue, 13 Dec 2011, Rory Mulvaney wrote:

> > On Mon, Dec 12, 2011 at 7:46 AM, Anthony R Fletcher <arif@mail.nih.gov>wrote:
> > 
> > > How can I wait for just the next job to finish?
> > >
> > 
> > One thing that may help you is TRAPCHLD.  Sadly, the signal handler doesn't
> > tell you what pid it is reacting to, nor the exit code.
> > 
> > TRAPCHLD() {
...
> > }
...
> > It might be nice to set an environment parameter with the pid and status
> > info right before the dotrap(SIGCHLD) call in jobs.c.
...

This sounds very promising, because you then could 'parallelize'
a loop by feeding it new processes when 'one of the runnig' dies.

I hope for something in the line of:

TRAPCHILD() { ... ... ... }
pids=( )
for i in some paramters given
do
        if [[ $#pids >= $MAXPARALLEL ]]
        then sleep 1 # and TRAPCHILD will remove 'done's
        else
                call $i &
                pids+=$!
        fi
done

SO one might be able to keep a bunch of CPUS working
for 'one given loop' without needing to know which
case is done first? (And only if the parallelities
do not break something like tempfiles, stderr, etc ...)

Stucki


-- 
Christoph von Stuckrad      * * |nickname |Mail <stucki@mi.fu-berlin.de> \
Freie Universitaet Berlin   |/_*|'stucki' |Tel(Mo.,Mi.):+49 30 838-75 459|
Mathematik & Informatik EDV |\ *|if online|  (Di,Do,Fr):+49 30 77 39 6600|
Takustr. 9 / 14195 Berlin   * * |on IRCnet|Fax(home):   +49 30 77 39 6601/

Re: wait for the next process to finish

[Bart Schaefer]

On Dec 13,  5:49pm, Christoph (Stucki) von Stuckrad wrote:
}
} This sounds very promising, because you then could 'parallelize'
} a loop by feeding it new processes when 'one of the runnig' dies.

That much you can do already.

    TRAPCHLD() {
       while (( ${#jobstates} < MAXPARALLEL ))
       do the background task &
       done
    }

(Of course you need some sort of other loop-ending condition, but you
get the idea.)

You don't really even need the trap for this, your original loop will
work fine as long as you examine ${(k)jobstates} as the list of PIDs
instead of trying to maintain your own array.

Re: wait for the next process to finish

[Rory Mulvaney]

On Tue, 13 Dec 2011, Rory Mulvaney wrote:

> On Mon, 12 Dec 2011, Wayne Davison wrote:
> 
> > On Mon, Dec 12, 2011 at 7:46 AM, Anthony R Fletcher <arif@mail.nih.gov>wrote:
> > 
> > > How can I wait for just the next job to finish?
> > >
> > 
> > One thing that may help you is TRAPCHLD.  Sadly, the signal handler doesn't
> > tell you what pid it is reacting to, nor the exit code.
> > 
> > TRAPCHLD() {
> >     echo here
> >     oldpids=($pids)
> >     pids=( )
> >     for p in $oldpids; do
> >         if kill -0 $p 2>/dev/null; then
> >             pids+=$p
> >         else
> >             #wait $p # Sadly, this doesn't work
> >             echo $p exited
> >         fi
> >     done
> > }
> > pids=( )
> > sleep 10 &
> > pids+=$!
> > sleep 20 &
> > pids+=$!
> > (sleep 15; false) &
> > pids+=$!
> > echo $pids
> > wait
> > echo done
> > 
> > It might be nice to set an environment parameter with the pid and status
> > info right before the dotrap(SIGCHLD) call in jobs.c.
> > 
> > ..wayne..
> > 
> 
> To clarify (I think this is fairly simple), you can supply the process id 
> as a parameter to 'wait', and though the $! method seems rather clumsy to 
> retrieve the pid (since you have to retrieve it somehow in the next 
> command after spawning the background process), it seems to work mostly in 
> general.
> 
> So you could do:
> 
> sleep 20000 &
> sleep 20 &
> pid=$!
> wait $pid
> 
> That will just wait for the sleep 20 process to complete while the sleep 
> 20000 process still runs in the background.
> 
> For further reference, I see that $! is documented in sec. 15.5 
> (Parameters set by the shell) in the zsh texinfo manual.
> 

Sorry, now I realized that you want to want to wait for whichever 
background process finishes first.  I know that select (zsh/zselect) is 
used for this type of thing, which blocks until some file descriptor is 
ready for reading or writing.  It would possible but sort of complicated 
to hack that functionality into the case of waiting for pids.  I can 
imagine using coproc to get a file descriptor for each of the background 
processes.

Something like this (requiring a zsh version with anonymous functions):

pids=( )
R=( $RANDOM $RANDOM )
echo $R
coproc () { sleep $(( $R[1]/32767.0*20 )) ; echo $$ }
pids+=$!
# copy the coproc fd
exec 3<&p
coproc () { sleep $(( $R[2]/32767.0*20 )) ; echo $$ }
pids+=$!  
exec 4<&p
zmodload zsh/zselect
zselect -r 3 4
echo $reply

Then one could parse $reply.

I don't know, maybe there's a much easier way to create file descriptors 
for those processes?

-Rory

Re: wait for the next process to finish

[Anthony R Fletcher]

On 13 Dec 2011 at 11:04:44, Rory Mulvaney wrote:
> pids=( )
> R=( $RANDOM $RANDOM )
> echo $R
> coproc () { sleep $(( $R[1]/32767.0*20 )) ; echo $$ }
> pids+=$!
> # copy the coproc fd
> exec 3<&p
> coproc () { sleep $(( $R[2]/32767.0*20 )) ; echo $$ }
> pids+=$!  
> exec 4<&p
> zmodload zsh/zselect
> zselect -r 3 4
> echo $reply
> 
> Then one could parse $reply.
> 
> I don't know, maybe there's a much easier way to create file descriptors 
> for those processes?
> 
> -Rory

I was using this loop

  for f in *.txt
  do
	  what-ever-has-to-happen $f &
	  pwait 20
  done

where pwait is a function

  function pwait() {
          while [ $(jobs -rp | wc -l) -ge $1 ]
          do
		  # sleep 10
                  wait
          done
  }

(see http://superuser.com/questions/158165/parallel-shell-loops for one
example amongst many).

I was hoping that using "wait" would be better than using a "sleep 10".
But wait waits for them all and not just he last to finish. The nice
thing is that this is a small, easy loop and all the brains is in the
barrier function pwait. Sadly a sleep for 10 seconds is completely
arbitrary.

			Anthony.

-- 
Anthony R Fletcher        
  Room 2033, Building 12A,        http://dcb.cit.nih.gov/~arif
  National Institutes of Health,  arif@mail.nih.gov
  12A South Drive, Bethesda,      Phone: (+1) 301 402 1741.
  MD 20892-5624, USA.

Re: wait for the next process to finish

[Jérémie Roquet]

2011/12/13 Anthony R Fletcher <arif@mail.nih.gov>:
> I was using this loop
>
>  for f in *.txt
>  do
>          what-ever-has-to-happen $f &
>          pwait 20
>  done
>
> where pwait is a function
>
>  function pwait() {
>          while [ $(jobs -rp | wc -l) -ge $1 ]
>          do
>                  # sleep 10
>                  wait
>          done
>  }
>
> (see http://superuser.com/questions/158165/parallel-shell-loops for one
> example amongst many).
>
> I was hoping that using "wait" would be better than using a "sleep 10".
> But wait waits for them all and not just he last to finish. The nice
> thing is that this is a small, easy loop and all the brains is in the
> barrier function pwait. Sadly a sleep for 10 seconds is completely
> arbitrary.

I'm wondering if you're not just looking for zargs :-)

autoload -U zargs
zargs -n 1 -P 20 *.txt -- what-ever-has-to-happen

Best regards,

-- 
Jérémie

Re: wait for the next process to finish

[Stephane Chazelas]

2011-12-13 19:04:12 +0100, Jérémie Roquet:
[...]
> I'm wondering if you're not just looking for zargs :-)
[...]

Or GNU parallel or pexec.

-- 
Stephane

Re: wait for the next process to finish

[Anthony R Fletcher]

On 13 Dec 2011 at 18:20:03, Stephane Chazelas wrote:
> 2011-12-13 19:04:12 +0100, Jérémie Roquet:
> [...]
> > I'm wondering if you're not just looking for zargs :-)
> [...]
> 
> Or GNU parallel or pexec.
> 

I considered GNU parallel but the loop seems do much easier for what I
wanted to do. pexec needs to be compiled - too hard for this job. The
loop idea was just too easy to use.

I didn't know about zargs but I'll play with it. Thanks.

		Anthony.

Re: wait for the next process to finish

[Peter Stephenson]

On Mon, 12 Dec 2011 10:46:01 -0500
Anthony R Fletcher <arif@mail.nih.gov> wrote:
> I just realised that the 'wait' command will either wait for specified
> jobs or all jobs and nothing in between. The manual says "If job is not
> given then all currently active child processes are waited for.".
> 
> So 
> 	sleep 30 &
> 	sleep 10 &
> 	sleep 30 &
> 	sleep 30 &
> 	wait
> waits for all the sleeps to finish.
> 
> How can I wait for just the next job to finish?

Certainly the shell internals don't help you here.  There's code to look
at a specific job, decide if it's still going, and exit when it isn't,
which is behind the wait builtin with an argument.  There's nothing to
loop over all jobs, decide what's still going, wait for something to
happen, then work out what it was and hence if it can stop waiting.

The reason waiting for all jobs works is simply that it waits for jobs
in order; if an earlier job exits first it gets handled by the signal
handler, but the shell foreground doesn't notice until the job it's
actually waiting for exits, then it carries on down the list and picks
up anything that's already exited.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

Re: wait for the next process to finish

[Bart Schaefer]

On Dec 13, 10:01am, Peter Stephenson wrote:
} Subject: Re: wait for the next process to finish
}
} On Mon, 12 Dec 2011 10:46:01 -0500
} Anthony R Fletcher <arif@mail.nih.gov> wrote:
} > I just realised that the 'wait' command will either wait for specified
} > jobs or all jobs and nothing in between. The manual says "If job is not
} > given then all currently active child processes are waited for.".
} > 
} > So 
} > 	sleep 30 &
} > 	sleep 10 &
} > 	sleep 30 &
} > 	sleep 30 &
} > 	wait
} > waits for all the sleeps to finish.
} > 
} > How can I wait for just the next job to finish?
} 
} Certainly the shell internals don't help you here.  There's code to look
} at a specific job, decide if it's still going, and exit when it isn't,
} which is behind the wait builtin with an argument.  There's nothing to
} loop over all jobs, decide what's still going, wait for something to
} happen, then work out what it was and hence if it can stop waiting.

What's "just the next job" mean here?

Wait until any one of the four jobs finishes, or wait until a specific
job (e.g., the first one started, and then the second one, and then
the third, etc.) finishes?

I think there have been some answers on this thread that assume one
meaning and some the other, and therefore are contradictory.  If it's
the first meaning (any job) then Wayne's TRAPCHLD suggestion is the
most helpful, but it's true the trap doesn't get useful information
about which child set it off, and on top of that the handler is not
called until the child is already removed from the job table, so even
if you could figure out which one it was you can't "wait" to get its
exit status (I don't think).

In the realm of ugly hacks, you could run each child as a coprocess
that ends with "print $?".  Stash away the coprocess file descriptor
each time you start one, and then when they're all running, start a
loop polling that list of descriptors with "read -t" until you find
one from which you can read.  What you read will be the exit status.

Re: wait for the next process to finish

[Rory Mulvaney]

On Tue, 13 Dec 2011, Bart Schaefer wrote:

> 
> In the realm of ugly hacks, you could run each child as a coprocess
> that ends with "print $?".  Stash away the coprocess file descriptor
> each time you start one, and then when they're all running, start a
> loop polling that list of descriptors with "read -t" until you find
> one from which you can read.  What you read will be the exit status.
> 

Yes, this is what I demonstrated in my second email, but the idea about 
using TRAPCHLD is probably easiest, since you could have your child 
processes print out any info about the process such as the return code, 
redirecting it to a single file descriptor owned by a designated 
information-relay coproc (which merely echo's its stdin to stdout), and 
then parse the output from the coproc in TRAPCHLD, which is apparently one 
of the available trap functions running whenever a signal arrives, since 
SIGCHLD is a signal from signal(7) (I didn't realize that before).

On Tue, 13 Dec 2011, Bart Schaefer wrote:

>
>     TRAPCHLD() {
>        while (( ${#jobstates} < MAXPARALLEL ))
>        do the background task &
>        done
>     }
>
> (Of course you need some sort of other loop-ending condition, but you
> get the idea.)
>
> You don't really even need the trap for this, your original loop will
> work fine as long as you examine ${(k)jobstates} as the list of PIDs
> instead of trying to maintain your own array.
>

Re: wait for the next process to finish

[Stephane Chazelas]

2011-12-12 10:46:01 -0500, Anthony R Fletcher:
> I just realised that the 'wait' command will either wait for specified
> jobs or all jobs and nothing in between. The manual says "If job is not
> given then all currently active child processes are waited for.".
> 
> So 
> 	sleep 30 &
> 	sleep 10 &
> 	sleep 30 &
> 	sleep 30 &
> 	wait
> waits for all the sleeps to finish.
> 
> How can I wait for just the next job to finish?
[...]

What about:

{
(sleep 4; echo A) &
(sleep 1; echo B) &
(sleep 8; echo C) &
(sleep 2; echo D) &
(sleep 7; echo E) &
(sleep 3; echo F) &
} | while read task; do
  date +"%T: $task finished"
done

17:44:17: B finished
17:44:18: D finished
17:44:19: F finished
17:44:20: A finished
17:44:23: E finished
17:44:24: C finished

-- 
Stephane