zargs error with some combinations of arguments

zargs error with some combinations of arguments

[Alan Wagner-Krankel]

I've run across some conditions that apparently trip up calls to
zargs. A simplified example:

zsh -f <<\eof
  vals=(${(l:79::x:):-{001..256}})
  print count:${#vals} length:${(c)#vals}
  autoload -Uz zargs
  zargs -- $vals -- print -l
eof

The output:
count:256 length:20479
zargs: cannot fit single argument within size limit

This error occurs when:
- the combined length of the arguments that are going to be used in a
  call is less than the max argument size (-s, default is 20480), and
- the length of the command string plus the arguments is greater than
  the -s size value.



It looks like zargs was set up this way to emulate the behavior of
gxargs from GNU findutils. With the right options, you can get a
similar error:

gxargs -L 256 -s 20480 -x echo <<<${(f)${(l:79::x:):-{001..256}}}

On my system this returns:
gxargs: argument list too long

I wasn't able to find an input that gave this result when using
the (large) defaults for gxargs.



On the other hand, POSIX seems to want the constructed command to fit
within all of the various size arguments, if I'm reading this snippet
in the spec correctly:
-s size
  Invoke utility using as many standard input arguments as possible
  yielding a command line length less than size (a positive decimal
  integer) bytes. Fewer arguments shall be used if:
  - The total number of arguments exceeds that specified by the -n
    option.
  - The total number of lines exceeds that specified by the -L option.



I've run into the zargs error a couple of times recently with
real-world data, so my system is now using a version that is hopefully
a bit closer to the POSIX behavior.  The diff listing is below.

Another option would have been to use a large value for -s, and
significantly reduce the likelihood of hitting a set of inputs that
result in an error.  That seems to be the gxargs approach.



Any thoughts? Is there something else I missed, or is a change like
I've described something that you want to consider for a future
version of zargs?

Thank you for reading this far. Hope I didn't make too many mistakes
in my first post to the list :).

Thanks,
Awk



diff --context=1 zsh.5.9/Functions/Misc/zargs ~/.zfunctions/zargs
*** zsh.5.9/Functions/Misc/zargs
--- ~/.zfunctions/zargs
***************
*** 178,180 ****

! local -i end c=0
  if [[ $eof == -(e|-eof) ]]; then ((end=ARGC+1))
--- 178,180 ----

! local -i end a c=0
  if [[ $eof == -(e|-eof) ]]; then ((end=ARGC+1))
***************
*** 249,250 ****
--- 249,257 ----

+ (( a = s - ${#:-"$command "} ))
+ if (( a <= 0 ))
+ then
+     print -u2 'zargs: value for max-chars must be >= command length'
+     return 1
+ fi
+
  l=${${${l##*-(l|L|-max-lines(=|))}[-1]}:-${${l[1]:+1}:-$ARGC}}
***************
*** 291,293 ****
    ((ARGC)) || break
!   for (( end=l; end && ${(c)#argv[1,end]} > s; end/=2 )) { }
    (( end > n && ( end = n ) ))
--- 298,300 ----
    ((ARGC)) || break
!   for (( end=l; end && ${(c)#argv[1,end]} > a; end/=2 )) { }
    (( end > n && ( end = n ) ))

Re: zargs error with some combinations of arguments

[Bart Schaefer]

On Thu, Aug 18, 2022 at 7:12 PM Alan Wagner-Krankel <awk@awkwork.com> wrote:
>
> count:256 length:20479
> zargs: cannot fit single argument within size limit
>
> + (( a = s - ${#:-"$command "} ))
> + if (( a <= 0 ))
> + then
> +     print -u2 'zargs: value for max-chars must be >= command length'
> +     return 1
> + fi

Isn't that just trading one error message for a different one?

The below would seem to make more sense, is there something I missed?

diff --git a/Functions/Misc/zargs b/Functions/Misc/zargs
index 782d6811e..2f5447917 100644
--- a/Functions/Misc/zargs
+++ b/Functions/Misc/zargs
@@ -305,7 +305,7 @@ do
     repeat $P
     do
     ((ARGC)) || break
-    for (( end=l; end && ${(c)#argv[1,end]} > s; end/=2 )) { }
+    for (( end=l; end && ${(c)#argv[1,end]} > s - $#command; end/=2 )) { }
     (( end > n && ( end = n ) ))
     args=( "${(@)argv[1,end]}" )
     shift $((end > ARGC ? ARGC : end))

Re: zargs error with some combinations of arguments

[Alan Wagner-Krankel]

I think we're both proposing basically the same change. It looks like
the format of my context diff hid the use of the calculated 'a' value
later in the code. Here's the diff listing from git:

diff --git a/Functions/Misc/zargs b/Functions/Misc/zargs
index 782d681..c7e96b8 100644
--- a/Functions/Misc/zargs
+++ b/Functions/Misc/zargs
@@ -194,7 +194,7 @@ else
     return 1
 fi

-local -i end c=0
+local -i end a c=0
 if [[ $eof == -(e|-eof) ]]; then ((end=ARGC+1))
 elif (( $#eof )); then end=$argv[(i)${eof##-(e|-eof=)}]
 else end=$argv[(i)--]
@@ -265,6 +265,13 @@ then
     return 1
 fi

+(( a = s - ${#:-"$command "} ))
+if (( a <= 0 ))
+then
+    print -u2 'zargs: value for max-chars must be >= command length'
+    return 1
+fi
+
 l=${${${l##*-(l|L|-max-lines(=|))}[-1]}:-${${l[1]:+1}:-$ARGC}}
 if (( l <= 0 ))
 then
@@ -305,7 +312,7 @@ do
     repeat $P
     do
        ((ARGC)) || break
-       for (( end=l; end && ${(c)#argv[1,end]} > s; end/=2 )) { }
+       for (( end=l; end && ${(c)#argv[1,end]} > a; end/=2 )) { }
        (( end > n && ( end = n ) ))
        args=( "${(@)argv[1,end]}" )
        shift $((end > ARGC ? ARGC : end))


In this line:
  (( a = s - ${#:-"$command "} ))
's' is the max size of the total command line. The calculated value
'a' represents the maximum number of characters that can be used for
arguments to the command. It's stored in a variable to avoid
recalculating it each time in the primary nested loop.

The 'command' variable is an array, and we also need a space for the
separator between the command and the parameters. ${#:-"$command "}
was my (possibly too cute) way of calculating the total length of the
command plus the space. This should give the same result:
  (( a = s - ${(c)#command} - 1 ))


The error message after the 'a <= 0' test is only going to occur
with some edge cases and nonsensical parameters, e.g.:

  zargs -s 10 -- a b c -- somelongcmd

It's not really needed. If the command is too large, the 'cannot fit
single argument within size limit' error message will do the job.


Thanks for looking into this,
Awk