I've committed "zargs" ...

I've committed "zargs" ...

[Bart Schaefer]

I got a bit carried away and implemented the entire set of options from
GNU xargs 4.1.  Rather than post the result here, I've just committed it.
I didn't write any doc for contrib.yo ... does anyone think I should?

Here's `zargs --help':

----------
Usage: zargs [options --] [input-args] [-- command [initial-args]]

If command and initial-args are omitted, "print -r --" is used.

Options:
--eof[=eof-str], -e[eof-str]
    Change the end-of-input-args string from "--" to eof-str.  If
    given as --eof=, an empty argument is the end; as --eof or -e,
    with no (or an empty) eof-str, all arguments are input-args.
--exit, -x
    Exit if the size (see --max-chars) is exceeded.
--help
    Print this summary and exit.
--interactive, -p
    Prompt before executing each command line.
--max-args=max-args, -n max-args
    Use at most max-args arguments per command line.
--max-chars=max-chars, -s max-chars
    Use at most max-chars characters per command line.
--max-lines[=max-lines], -l[max-lines]
    Use at most max-lines of the input-args per command line.
    This option is misnamed for xargs compatibility.
--max-procs=max-procs, -P max-procs
    Run up to max-procs command lines in the background at once.
--no-run-if-empty, -r
    Do nothing if there are no input arguments before the eof-str.
--null, -0
    Split each input-arg at null bytes, for xargs compatibility.
--replace[=replace-str], -i[replace-str]
    Substitute replace-str in the initial-args by each initial-arg.
    Implies --exit --max-lines=1.
--verbose, -t
    Print each command line to stderr before executing it.
--version
    Print the version number of zargs and exit.
----------

I spent much more time than I should have on getting the best --max-procs
algorithm I could come up with.  Comments welcome.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Occasional 'job table full or recursion limit exceeded'

[Felix Rosencrantz]

--- Bart Schaefer <schaefer@brasslantern.com> wrote:

> I spent much more time than I should have on getting the best
> --max-procs algorithm I could come up with.  Comments welcome.

Not really sure if this is the feedback you want.  Since I mostly
only see it in long running shells, where my suspended processes have
accrued, it seems like increasng it by small or fractional increments
would be more conservative of memory usage than a doubling strategy.

When I see the problem I typically have 10 to 15 processes.  It's unlikely
I would have much more than 20, because that is getting in the range where it
is difficult to manage.

-FR.



__________________________________________________
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com

Re: Occasional 'job table full or recursion limit exceeded'

[Bart Schaefer]

On Sep 9,  7:41am, Felix Rosencrantz wrote:
} Subject: Re: Occasional 'job table full or recursion limit exceeded'
}
} --- Bart Schaefer <schaefer@brasslantern.com> wrote:
} 
} > I spent much more time than I should have on getting the best
} > --max-procs algorithm I could come up with.  Comments welcome.
} 
} Not really sure if this is the feedback you want.

I'm not really sure what to say, because the message you quoted doesn't
have anything to do with the thread you're replying to.  (The excerpt
above refers to the number of background jobs spawned by my "zargs"
function, not to the value of MAXJOBS in the C code.)

} Since I mostly only see it in long running shells, where my suspended
} processes have accrued, it seems like increasng it by small or
} fractional increments would be more conservative of memory usage than
} a doubling strategy.

Part of the problem is that the value of MAXJOBS was established when
each job table entry represented a real unix process.  Now that we make
job table entries for internal shell control structures, we need more
table space.  It still shouldn't need to grow without bound (though I
believe we have other anti-infinite recursion mechanisms in place now).

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Occasional 'job table full or recursion limit exceeded'

[Felix Rosencrantz]

--- Bart Schaefer <schaefer@brasslantern.com> wrote:
> I'm not really sure what to say, because the message you quoted doesn't
> have anything to do with the thread you're replying to.  (The excerpt
> above refers to the number of background jobs spawned by my "zargs"
> function, not to the value of MAXJOBS in the C code.)
I'm a wee bit red faced... :)

> Part of the problem is that the value of MAXJOBS was established when
> each job table entry represented a real unix process.  Now that we make
> job table entries for internal shell control structures, we need more
> table space.  It still shouldn't need to grow without bound (though I
> believe we have other anti-infinite recursion mechanisms in place now).
I didn't really appreciate that MAXJOBS is sort of taking on the role of a
control structures stack.   In which case, a dynamic solution seems reasonable.

Currently I'm building with MAXJOBS doubled, figuring that should cover it for
me.   Though I suspect most users have prebuilt versions of zsh, and so might
hit this problem.

-FR.

__________________________________________________
Yahoo! - We Remember
9-11: A tribute to the more than 3,000 lives lost
http://dir.remember.yahoo.com/tribute

Occasional 'job table full or recursion limit exceeded'

[Felix Rosencrantz]

There are times I will see the following error in my long running active
shells:
	_description:70: job table full or recursion limit exceeded

I only see this message when attempting to perform completions.  The usual
place I hit the problem is with filename completion on cvs commands.  Though I
will see this problem elsewhere (like this one was when I was completing
'./configure --prefix=<TAB>').  The actual function hitting the problem will
change from each attempt at calling completion.

My job table is not full.  This is a completion I regularly perform without a
problem.  Also, it is possible later in the same shell to have this problem
cleared.   Though once I hit the problem, I can reproduce it reliably, the
completion will always fail, until it clears itself (if it does clear).   (Not
sure what is clearing it.)


This problem was been around for some time, so it's not due to recent changes. 
Not sure if it is the job table or completion.

Any thoughts?

-FR.



__________________________________________________
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com

Re: Occasional 'job table full or recursion limit exceeded'

[Bart Schaefer]

On Aug 28, 12:31am, Felix Rosencrantz wrote:
} Subject: Occasional 'job table full or recursion limit exceeded'
}
} There are times I will see the following error in my long running active
} shells:
} 	_description:70: job table full or recursion limit exceeded
} 
} My job table is not full.

The message is slightly misleading.  Many built-in shell constructs (such
as "while" and other loops) allocate job table entries even when running
in the current shell, for purposes of simulated signal handling (e.g., so
you can stop the entire loop with ^Z).  That's what the "recursion limit"
part is talking about.

Since much of the completion system is implemented as nested loops over
the set of tags, the set of possible matches, etc., it's conceivable that
some completions are running out of job table entries.  The size of the
table is only 50 unless `configure --enable-max-jobtable-size' is used,
and (having seen a number of _complete_debug call traces) I can believe
that you'd reach that.

We should probably consider increasing the default MAXJOB value.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Occasional 'job table full or recursion limit exceeded'

[Peter Stephenson]

"Bart Schaefer" wrote:
> We should probably consider increasing the default MAXJOB value.

Maybe there's some mileage in allocating the table at run time?  We would
still need a maximum but it could be larger without affecting most users.
Since we usually look it up by number, reallocating it shouldn't be a
particular problem --- we already effectively do this for the `jobs |'
hac^H^H^Hworkaround.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************

Re: Occasional 'job table full or recursion limit exceeded'

[Peter Stephenson]

Peter Stephenson wrote:
> "Bart Schaefer" wrote:
> > We should probably consider increasing the default MAXJOB value.
> 
> Maybe there's some mileage in allocating the table at run time?  We would
> still need a maximum but it could be larger without affecting most users.
> Since we usually look it up by number, reallocating it shouldn't be a
> particular problem --- we already effectively do this for the `jobs |'
> hac^H^H^Hworkaround.

(Plus the number could be made configurable at run time.)

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************