[PATCH] Removed arbitrary limitations on array accesses

[PATCH] Removed arbitrary limitations on array accesses

[Michael Hwang]

This issue was brought up on IRC. It appears that while there is no limit on
how many array elements can be stored, there is a limit to how many can be
accessed. This patch removes these limits.

Michael Hwang

---
 Src/params.c |   16 ----------------
 Src/zsh.h    |    2 --
 2 files changed, 0 insertions(+), 18 deletions(-)

diff --git a/Src/params.c b/Src/params.c
index 0425e07..d092ccc 100644
--- a/Src/params.c
+++ b/Src/params.c
@@ -1853,22 +1853,6 @@ fetchvalue(Value v, char **pptr, int bracks, int flags)
     if (!bracks && *s)
 	return NULL;
     *pptr = s;
-    if (v->start > MAX_ARRLEN) {
-	zerr("subscript too %s: %d", "big", v->start + !isset(KSHARRAYS));
-	return NULL;
-    }
-    if (v->start < -MAX_ARRLEN) {
-	zerr("subscript too %s: %d", "small", v->start);
-	return NULL;
-    }
-    if (v->end > MAX_ARRLEN+1) {
-	zerr("subscript too %s: %d", "big", v->end - !!isset(KSHARRAYS));
-	return NULL;
-    }
-    if (v->end < -MAX_ARRLEN) {
-	zerr("subscript too %s: %d", "small", v->end);
-	return NULL;
-    }
     return v;
 }
 
diff --git a/Src/zsh.h b/Src/zsh.h
index 060f8a6..1b407b0 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -626,8 +626,6 @@ enum {
     VALFLAG_SUBST =     0x0004  /* Substitution, so apply padding, case flags */
 };
 
-#define MAX_ARRLEN    262144
-
 /********************************************/
 /* Definitions for word code                 */
 /********************************************/
-- 
1.6.2.5

Re: [PATCH] Removed arbitrary limitations on array accesses

[Peter Stephenson]

On Mon,  4 Jan 2010 20:38:17 -0500
Michael Hwang <michael.a.hwang@gmail.com> wrote:
> This issue was brought up on IRC. It appears that while there is no limit on
> how many array elements can be stored, there is a limit to how many can be
> accessed. This patch removes these limits.

Those have been there for a long time.  I don't have any evidence that
they're doing a lot of good but we have had people creating positional
parameters with <long_number>=something and wondering why it uses a lot of
memory.  I suppose this is similar.  The arbitrary limit is not very
useful and also undocumented; most people wouldn't miss it if it wasn't
there, certainly.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: [PATCH] Removed arbitrary limitations on array accesses

[Bart Schaefer]

On Jan 5,  9:48am, Peter Stephenson wrote:
}
} Those have been there for a long time. I don't have any evidence that
} they're doing a lot of good but we have had people creating positional
} parameters with <long_number>=something and wondering why it uses a
} lot of memory. I suppose this is similar. The arbitrary limit is not
} very useful and also undocumented; most people wouldn't miss it if it
} wasn't there, certainly.

I have a vague memory of when this was added, but it appears to have been
sometime longer ago than when the zsh-workers archive begins and after
the last zsh-2 archive I kept -- which means it was late 1994 or early
1995.

My recollection, such as it is, is that attempting to read an array
slice from N to some huge number would allocate a temporary array with
empty slots for all the intervening positions, causing the shell to
crash with an out-of-memory error or to DoS attack the university time-
share system by attempting to consume all memory.  The arbitrary 262144
number was chosen based on some computation of the space occupied by
one of those empty slots, to assure that the resulting temporary would
be smaller than the typical VM limitations of a machine of the era.

It seems like we need SOME kind of limit here to prevent the user from
accidentally consuming huge amounts of memory, but it's probably useful
for it to become a computed value based on process limits.

Re: [PATCH] Removed arbitrary limitations on array accesses

[Mikael Magnusson]

2010/1/5 Bart Schaefer <schaefer@brasslantern.com>:
> On Jan 5,  9:48am, Peter Stephenson wrote:
> }
> } Those have been there for a long time. I don't have any evidence that
> } they're doing a lot of good but we have had people creating positional
> } parameters with <long_number>=something and wondering why it uses a
> } lot of memory. I suppose this is similar. The arbitrary limit is not
> } very useful and also undocumented; most people wouldn't miss it if it
> } wasn't there, certainly.
>
> I have a vague memory of when this was added, but it appears to have been
> sometime longer ago than when the zsh-workers archive begins and after
> the last zsh-2 archive I kept -- which means it was late 1994 or early
> 1995.
>
> My recollection, such as it is, is that attempting to read an array
> slice from N to some huge number would allocate a temporary array with
> empty slots for all the intervening positions, causing the shell to
> crash with an out-of-memory error or to DoS attack the university time-
> share system by attempting to consume all memory.  The arbitrary 262144
> number was chosen based on some computation of the space occupied by
> one of those empty slots, to assure that the resulting temporary would
> be smaller than the typical VM limitations of a machine of the era.
>
> It seems like we need SOME kind of limit here to prevent the user from
> accidentally consuming huge amounts of memory, but it's probably useful
> for it to become a computed value based on process limits.

echo $path[1,100000000000000000] appears to work fine here (though
$path[1,1000000000000000000] seems to expand to nothing) without using
any noticable memory. If you mean an array that is actually that size,
then the memory is already used, so the worst would be doubling what
is already used? If it only affects slices, could single element
access still be unlimited?

I notice that even before the patch, using [-n]  to access the last
element seems to work for larger arrays.

% test=({1..300000})
% echo $test[270000]
zsh: subscript too big: 270000
% echo $test[-1]
300000
% echo $test[-2]
299999
% echo $test[-100]
299901
% echo $test[-100,-90]
299901 299902 299903 299904 299905 299906 299907 299908 299909 299910 299911
% echo $test[-270000]
zsh: subscript too small: -270000

-- 
Mikael Magnusson

Re: [PATCH] Removed arbitrary limitations on array accesses

[Geoff Wing]

On Tuesday 2010-01-05 07:32 -0800, Bart Schaefer output:
:I have a vague memory of when this was added, but it appears to have been
:sometime longer ago than when the zsh-workers archive begins and after
:the last zsh-2 archive I kept -- which means it was late 1994 or early
:1995.

Appeared in 2.5.03:

Fri Jul 29 20:24:07 1994  Richard Coleman
	* params.c: additional patch by Sven for safeguarding
	  size of array subscripts.

Fri Jul 29 20:06:24 1994  Richard Coleman
	* params.c,zsh.h: patch by Sven to add safeguard to size of
	  array subscripts.

I'm pretty sure that I've lost all that mail and I can't find references
to it on any search engines either.  What's the reason for comments in
code again? :-)

Regards,
Geoff

Re: [PATCH] Removed arbitrary limitations on array accesses

[Duncan Sinclair]

On 5 Jan 2010, at 9:48 am, Peter Stephenson wrote:

> On Mon,  4 Jan 2010 20:38:17 -0500
> Michael Hwang <michael.a.hwang@gmail.com> wrote:
>> This issue was brought up on IRC. It appears that while there is no  
>> limit on
>> how many array elements can be stored, there is a limit to how many  
>> can be
>> accessed. This patch removes these limits.
>
> Those have been there for a long time.  I don't have any evidence that
> they're doing a lot of good but we have had people creating positional
> parameters with <long_number>=something and wondering why it uses a  
> lot of
> memory.  I suppose this is similar.  The arbitrary limit is not very
> useful and also undocumented; most people wouldn't miss it if it  
> wasn't
> there, certainly.

IIRC, the problem was that is you typed a big number at the zsh prompt  
and hit tab, the shell would either hang for a long time or crash (out  
of memory).

% 99999999999<tab>

I reported this as a bug – the shell should not crash so easily.

There is probably a way of preventing this problem though without  
putting arbitrary limits on the size of array indices.


-- 
Duncan Sinclair  |  {+44|0}141 548 3592  |  cis.strath.ac.uk
System Administrator
Computer and Information Sciences
University of Strathclyde
The University of Strathclyde is a charitable body, registered in  
Scotland, number SC015263.

Re: [PATCH] Removed arbitrary limitations on array accesses

[Mikael Magnusson]

2010/1/6 Duncan Sinclair <duncan.sinclair@cis.strath.ac.uk>:
>
> On 5 Jan 2010, at 9:48 am, Peter Stephenson wrote:
>
>> On Mon,  4 Jan 2010 20:38:17 -0500
>> Michael Hwang <michael.a.hwang@gmail.com> wrote:
>>>
>>> This issue was brought up on IRC. It appears that while there is no limit
>>> on
>>> how many array elements can be stored, there is a limit to how many can
>>> be
>>> accessed. This patch removes these limits.
>>
>> Those have been there for a long time.  I don't have any evidence that
>> they're doing a lot of good but we have had people creating positional
>> parameters with <long_number>=something and wondering why it uses a lot of
>> memory.  I suppose this is similar.  The arbitrary limit is not very
>> useful and also undocumented; most people wouldn't miss it if it wasn't
>> there, certainly.
>
> IIRC, the problem was that is you typed a big number at the zsh prompt and
> hit tab, the shell would either hang for a long time or crash (out of
> memory).
>
> % 99999999999<tab>
>
> I reported this as a bug – the shell should not crash so easily.
>
> There is probably a way of preventing this problem though without putting
> arbitrary limits on the size of array indices.

There are other similar ways you can crash a shell anyway,
alt-111111111111 1 and `yes` for example. I'm not sure if it's worth
trying to fix all of them.

-- 
Mikael Magnusson

Re: [PATCH] Removed arbitrary limitations on array accesses

[Richard Hartmann]

On Wed, Jan 6, 2010 at 15:02, Mikael Magnusson <mikachu@gmail.com> wrote:
> 2010/1/6 Duncan Sinclair <duncan.sinclair@cis.strath.ac.uk>:

>> % 99999999999<tab>
>>
>> I reported this as a bug – the shell should not crash so easily.
>
> There are other similar ways you can crash a shell anyway,
> alt-111111111111 1 and `yes` for example. I'm not sure if it's worth
> trying to fix all of them.

Completing on arbitrary input is a lot more likely than your
example, though.


Richard