job control problem

job control problem

[hanpingtian]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

 Hello,
I feel zsh's job control is a little weak than bash. For example, this script:

% cat t.sh
for ((i=1;i<=10000;i++))
do
    sleep 1 &
done

wait

With zsh:
% zsh -x ./t.sh
...
./t.sh:3: fork failed: resource temporarily unavailable
+./t.sh:6> wait
+./t.sh:3> sleep 1
it will fail with the fork() failure very soon.

But with bash:
% bash -x ./t.sh
...
+ (( i++ ))
+ (( i<=10000 ))
./t.sh: fork: retry: Resource temporarily unavailable
+ sleep 1
+ sleep 1
+ sleep 1
+ sleep 1
+ sleep 1
+ sleep 1
+ sleep 1
+ sleep 1
./t.sh: fork: retry: Resource temporarily unavailable
+ (( i++ ))
...
It can run more longer by retry and retry.
And I noticed that when trying to run a lot of jobs in background can cause zsh quit or frozen.

Re: job control problem

[Bart Schaefer]

On Mar 6,  6:01pm, hanpingtian wrote:
}
} And I noticed that when trying to run a lot of jobs in background can
} cause zsh quit or frozen.

Hmm.

Zsh is attempting to reclaim job table space during the SIGCLD handler.
I did a quick test and somewhere after a few hundred jobs the child
handler gets called during a free(), and the handler then calls free()
again and ends up deadlocked on a system mutex in the guts of glibc.

This is especially likely with "zsh -x" because the PS4 prompt is
allocated and freed each time it's printed.  If I leave off the -x,
the whole loop to 10000 runs to completion.

However, if I try to reproduce this with bash by frobbing "ulimit -u",
bash exits for me at exactly the same point as zsh.  In fact we just
had a whole discussion on zsh-workers [*] about bash exiting on fork
failure, and how zsh either did not but ought to, or exited success
when it should probably exit with failure.

I modified t.sh to echo the loop counter so I cound run it without -x
and still see how many iterations it made.

schaefer[501] bash --version
GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin11)
Copyright (C) 2007 Free Software Foundation, Inc.
schaefer[502] bash /tmp/t.sh
1
...
67
/tmp/t.sh: fork: Resource temporarily unavailable
schaefer[503] echo $?
128

No "retry" mentioned.  What am I missing?


[*] http://www.zsh.org/mla/workers/2012/msg00187.html thread up to
    http://www.zsh.org/mla/workers/2012/msg00201.html or so

Re: job control problem

[Chet Ramey]

On 3/7/12 12:46 AM, Bart Schaefer wrote:

> schaefer[501] bash --version
> GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin11)
> Copyright (C) 2007 Free Software Foundation, Inc.
> schaefer[502] bash /tmp/t.sh
> 1
> ...
> 67
> /tmp/t.sh: fork: Resource temporarily unavailable
> schaefer[503] echo $?
> 128
> 
> No "retry" mentioned.  What am I missing?

Bash-4.2 will sleep for 1, 2, 4, then 8 seconds if a fork fails, trying
after each sleep to reap dead children.  It will give up after that and
report the fork failure.  You'll see messages like this:

$ ls
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: Resource temporarily unavailable

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    chet@case.edu    http://cnswww.cns.cwru.edu/~chet/

Re: job control problem

[Bart Schaefer]

On Mar 7,  8:45am, Chet Ramey wrote:
}
} Bash-4.2 will sleep for 1, 2, 4, then 8 seconds if a fork fails, trying
} after each sleep to reap dead children.  It will give up after that and
} report the fork failure.

Hmm, that seems like it could be a problem in some rare cases.  Is there
a way to turn this off?

Re: job control problem

[Chet Ramey]

On 3/7/12 12:48 PM, Bart Schaefer wrote:
> On Mar 7,  8:45am, Chet Ramey wrote:
> }
> } Bash-4.2 will sleep for 1, 2, 4, then 8 seconds if a fork fails, trying
> } after each sleep to reap dead children.  It will give up after that and
> } report the fork failure.
> 
> Hmm, that seems like it could be a problem in some rare cases.  Is there
> a way to turn this off?

You can recompile bash with FORKSLEEP_MAX = 0 if you want to disable it.
What rare cases were you thinking of?

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    chet@case.edu    http://cnswww.cns.cwru.edu/~chet/

Re: job control problem

[Bart Schaefer]

} > } Bash-4.2 will sleep for 1, 2, 4, then 8 seconds if a fork fails, trying
} > } after each sleep to reap dead children.
} > 
} > Hmm, that seems like it could be a problem in some rare cases.
} 
} What rare cases were you thinking of?

Any in which the shell is itself fork-bombing the system.  Sure, you can
always work out a way to do so intentionally, but the shell retrying on
failure would seem to make it way too easy to do it accidentally.

Re:Re: job control problem

[hanpingtian]

On Tue, Mar 06, 2012 at 09:46:08PM -0800, Bart Schaefer wrote:
> On Mar 6,  6:01pm, hanpingtian wrote:
> }
> } And I noticed that when trying to run a lot of jobs in background can
> } cause zsh quit or frozen.
> 
> Hmm.
> 
> Zsh is attempting to reclaim job table space during the SIGCLD handler.
> I did a quick test and somewhere after a few hundred jobs the child
> handler gets called during a free(), and the handler then calls free()
> again and ends up deadlocked on a system mutex in the guts of glibc.

I noticed that sometimes zsh scripts hangs up at futex(), after run some
background jobs.

> 
> This is especially likely with "zsh -x" because the PS4 prompt is
> allocated and freed each time it's printed.  If I leave off the -x,
> the whole loop to 10000 runs to completion.
> 
> However, if I try to reproduce this with bash by frobbing "ulimit -u",
> bash exits for me at exactly the same point as zsh.  In fact we just
> had a whole discussion on zsh-workers [*] about bash exiting on fork
> failure, and how zsh either did not but ought to, or exited success
> when it should probably exit with failure.
> 
> I modified t.sh to echo the loop counter so I cound run it without -x
> and still see how many iterations it made.
> 
> schaefer[501] bash --version
> GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin11)
> Copyright (C) 2007 Free Software Foundation, Inc.
> schaefer[502] bash /tmp/t.sh
> 1
> ...
> 67
> /tmp/t.sh: fork: Resource temporarily unavailable
> schaefer[503] echo $?
> 128
> 
> No "retry" mentioned.  What am I missing?
I was using bash 4.2.10 to do this testing:
% bash --version
GNU bash, version 4.2.10(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>

Looks like the bash will retry again and again.

> 
> 
> [*] http://www.zsh.org/mla/workers/2012/msg00187.html thread up to
>     http://www.zsh.org/mla/workers/2012/msg00201.html or so