Length of argument list

Length of argument list

[Manfred Lotz]

Hi all,
I created 1 million files in a directory callend manyfiles/.

Now 
ls manyfiles/* |wc -l

gives 
zsh: argument list too long: ls


Question: Is there a way to change the maximum size of the argument
list?



-- 
Manfred

Re: Length of argument list

[Mikael Magnusson]

On Wed, Jun 10, 2015 at 8:48 PM, Manfred Lotz <manfred.lotz@arcor.de> wrote:
> Hi all,
> I created 1 million files in a directory callend manyfiles/.
>
> Now
> ls manyfiles/* |wc -l
>
> gives
> zsh: argument list too long: ls
>
>
> Question: Is there a way to change the maximum size of the argument
> list?

That depends on your operating system, on linux it is guided by ulimit
-s (in particular, the argument list is a quarter of the stack size).
On other operating systems, the limit is traditionally quite small.
I'm assuming the above is just an example, but that particular thing
would be better handled by () { echo $# } manyfiles/*, or some
xargs/zargs contraption if you want to be more general.

-- 
Mikael Magnusson

Re: Length of argument list

[Manfred Lotz]

On Wed, 10 Jun 2015 22:09:06 +0200
Mikael Magnusson <mikachu@gmail.com> wrote:

> On Wed, Jun 10, 2015 at 8:48 PM, Manfred Lotz <manfred.lotz@arcor.de>
> wrote:
> > Hi all,
> > I created 1 million files in a directory callend manyfiles/.
> >
> > Now
> > ls manyfiles/* |wc -l
> >
> > gives
> > zsh: argument list too long: ls
> >
> >
> > Question: Is there a way to change the maximum size of the argument
> > list?
> 
> That depends on your operating system, on linux it is guided by ulimit
> -s (in particular, the argument list is a quarter of the stack size).

Interesting. Didn't know. This works indeed.

Do you know how it is in FreeBSD, for instance?

> On other operating systems, the limit is traditionally quite small.
> I'm assuming the above is just an example, but that particular thing

Yes, just an example. I want to see if it possible to use simple
command when dealing with directories containing many files.


> would be better handled by () { echo $# } manyfiles/*, or some
> xargs/zargs contraption if you want to be more general.
> 

..and for f in *... and things like this.


In the end it means if a directory contains a large number of files
then the usual commands rm, cp, mv etc cannot be used easily without
additional support (find,xargs, for loops etc).

-- 
Manfred

Re: Length of argument list

[Clint Hepner]

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

This is the wrong way to count files in the first place; any matching file
names containing newlines (yes, they're legal) would give the wrong count.

One alternative is

    a=( manyfiles/* )
    echo ${#a}

to create an array containing all the file names, then check the number of
elements in the array.

On Wed, Jun 10, 2015 at 2:48 PM, Manfred Lotz <manfred.lotz@arcor.de> wrote:

> Hi all,
> I created 1 million files in a directory callend manyfiles/.
>
> Now
> ls manyfiles/* |wc -l
>
> gives
> zsh: argument list too long: ls
>
>
> Question: Is there a way to change the maximum size of the argument
> list?
>
>
>
> --
> Manfred
>

Re: Length of argument list

[Manfred Lotz]

On Wed, 10 Jun 2015 17:46:24 -0400
Clint Hepner <clint.hepner@gmail.com> wrote:

> This is the wrong way to count files in the first place; any matching

As said already this was an example.


> file names containing newlines (yes, they're legal) would give the
> wrong count.
> 

I know but when dealing with files I never take into account
that file names contain newlines. IMHO, only crazy people create file
names which contain newlines.


-- 
Manfred

Re: Length of argument list

[Stephane Chazelas]

2015-06-11 11:27:16 +0200, Manfred Lotz:
[...]
> I know but when dealing with files I never take into account
> that file names contain newlines. IMHO, only crazy people create file
> names which contain newlines.
[...]

Crazy, careless (copy-paste in a terminal) or malicious.

There are lots of security vulnerabilties around that are caused
by code that assumes file names cannot contain newline
characters.

-- 
Stephane