From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 12220 invoked by alias); 26 Mar 2011 14:04:48 -0000 Mailing-List: contact zsh-users-help@zsh.org; run by ezmlm Precedence: bulk X-No-Archive: yes List-Id: Zsh Users List List-Post: List-Help: X-Seq: 15904 Received: (qmail 22582 invoked from network); 26 Mar 2011 14:04:35 -0000 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on f.primenet.com.au X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_PASS autolearn=ham version=3.3.1 Received-SPF: pass (ns1.primenet.com.au: SPF record at myproxylists.com designates 217.119.39.74 as permitted sender) X-Originating-IP: 127.0.0.1 Message-ID: In-Reply-To: <110326001415.ZM31882@torch.brasslantern.com> References: <9861b13e6828d43041a5b489f016664a.squirrel@gameframe.net> <110325150350.ZM30913@torch.brasslantern.com> <9cf85fb41e83cdaa17ad01277e478571.squirrel@gameframe.net> <110325191023.ZM31052@torch.brasslantern.com> <110326001415.ZM31882@torch.brasslantern.com> Date: Sat, 26 Mar 2011 16:04:28 +0200 Subject: Re: How to add string to end of each array element without loops From: nix@myproxylists.com To: "Bart Schaefer" Cc: zsh-users@zsh.org User-Agent: SquirrelMail/1.4.20 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal > On Mar 26, 4:39am, nix@myproxylists.com wrote: > } > } Line 143: print -l ${(n)PROXIES} >> $Proxy_List causes segfault. I can't > } see where the problem is, if you try IP range up to one million, it will > } work good. > > Aha. I'm able to reproduce this with > > typeset -A a > a=({5000000..1} {1..5000000}) > print -l $a >> /dev/null > > What's actually failing is builtin.c line 283, the declaration > > VARARR(char *, argarr, argc + 1); > > This is attempting to allocate the entire argument list of "print" > (which in my example is 5 million words) on the C function call > stack. Even with "unlimit stacksize" this is likely to overflow. > The program merely doesn't discover that this has failed until a > few lines later when it tries to use the first element of the array. > I actually tried to double the limit for stacksize 'limit stacksize 32' but it did not fixed it. Looks like the limit is something like one million for 'print -l $a >> /dev/null'. > There's really nothing that can be done about this. The whole shell > paradigm of passing arguments as an argv[] array means that $a has > to be expanded and then "print" called with the words by value. > > So what you have to do to work around this is NOT rely on "print -l" > to insert the newlines, and instead insert them yourself: > > print ${(F)PROXIES} >> $Proxy_List > > This passes a single giant word with embedded newlines to "print" and > thus avoids allocating all that space on the C stack. > > You're really past the design limits here of what a language with the > semantic rules of a command shell is meant to deal with. If "print" > were not a builtin you'd have blown out the limits of argument passing > before even getting as far as you did; even with the (F) trick, using > /bin/echo will fail with "argument list too long". > ./SUBNET [+] Generating IP-addresses ... please wait [+] 2072640 IP-addresses has been generated ... [+] Generating proxy list ... please wait [+] 6217920 proxies has been generated. The (F} trick fixed it. Im quite confident that nearly none else expect me need that big arrays. 'SUBNET' script is very fast, it's actually able to beat in terms of speed solarwinds's advanced subnet calculator when it comes to generating IPs out of CIDR :) As I said earlier, my C programming skills are very limited but im getting quite good with ZSH and PHP. Mixing two powerful scripting languages (or programming languages) you name it, the results are quite powerful. > I noticed you loaded the mapfile module even though you don't use it. > It should work to do > > mapfile[$Proxy_list]=${(F)PROXIES} > > but whether that's actually faster than "print >>" I haven't checked. > I load other modules as well because most of my other tools are using those modules. I tried also with mapfile but it did segfaulted as well. Afterwards I noticed it's not even needed in 'SUBNET' script. Thanks a bunch again. Looks like I have found my ZSH guru :)