From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 25143 invoked by alias); 25 Mar 2011 01:12:18 -0000 Mailing-List: contact zsh-users-help@zsh.org; run by ezmlm Precedence: bulk X-No-Archive: yes List-Id: Zsh Users List List-Post: List-Help: X-Seq: 15893 Received: (qmail 11503 invoked from network); 25 Mar 2011 01:12:06 -0000 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on f.primenet.com.au X-Spam-Level: X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,SPF_HELO_PASS, URIBL_RHS_DOB autolearn=no version=3.3.1 Received-SPF: pass (ns1.primenet.com.au: SPF record at myproxylists.com designates 217.119.39.74 as permitted sender) X-Originating-IP: 127.0.0.1 Message-ID: <446d1efa4ce9f5b6bc6dc8782050cb3f.squirrel@gameframe.net> In-Reply-To: References: <5c830d92034e61a01c96e6aa4d10798c.squirrel@gameframe.net> Date: Fri, 25 Mar 2011 03:12:06 +0200 Subject: Re: Why large arrays are extremely slow to handle? From: nix@myproxylists.com To: "Mikael Magnusson" Cc: zsh-users@zsh.org User-Agent: SquirrelMail/1.4.20 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal > On 25 March 2011 01:37, wrote: >> Tested on AMD Phenom(tm) II X6 1090T Processor 3.6GHz using one core. >> >> I think there's is a big flaw somewhere that causes the following: >> >> #!/bin/zsh >> >> emulate zsh >> >> TEST=() >> >> for i in {1..10000} ; do >> >> TEST+="$i" # append (push) to an array >> >> done >> >> --- 10K >> time ./bench >> real 0m3.944s >> >> --- 50K BOOOM! WTF? >> >> time ./bench >> real 1m53.321s >> >> Does not make much sense to me. Im also a PHP developer. Just for >> comparison, let's do the same with PHP. >> >> > >> $test = array(); >> >> for ($i=1; $i < 50000; $i++) { >> >> $test[] = $i; >> >> } >> >> print_r($test); >> >> ?> >> >> --- 10K >> >> time php TEST_PHP >> real 0m0.011s >> >> --- 50K >> >> time php TEST_PHP >> real 0m0.025s >> >> >> Any ideas why it's extremely slow? I have need to use very large arrays >> (even over one million elements in a single array) but it's currently >> impossible due to the above. > > The problem is not the array, but that you are handing 50000 arguments > to the for loop. With this optimization it "only" takes 5 seconds ;) > for (( i = 0; i < 10000; i++ )) { arr+=$i } > That said, you generally don't want to use large arrays in zsh, it will be > slow. > > -- > Mikael Magnusson > There problem is the array. I tried it also on a DUAL X5450 XEON machine, terribly slow on it as well while using ZSH. I would love to have a fix. I just coded subnet generator in ZSH and noticed when i started to generate larger IP-ranges, things started to go very slowly :( Mikael, try this with provided example: arr=( $(print -r -- ${(u)=arr}) ) # List only unique elements in an array It's terrible slow as well with 50K elements, the problem is nothin but the array handling. #!/bin/zsh emulate zsh TEST=() for (( i = 0; i < 50000; i++ )) ; do TEST+="$i" done time ./BENC real 1m54.353s Not difference at all to "for {1..50000}" ;)