From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 25390 invoked by alias); 22 Nov 2009 22:27:25 -0000 Mailing-List: contact zsh-users-help@zsh.org; run by ezmlm Precedence: bulk X-No-Archive: yes List-Id: Zsh Users List List-Post: List-Help: X-Seq: 14592 Received: (qmail 11542 invoked from network); 22 Nov 2009 22:27:21 -0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on f.primenet.com.au X-Spam-Level: X-Spam-Status: No, score=-2.5 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.2.5 Received-SPF: none (ns1.primenet.com.au: domain at math.technion.ac.il does not designate permitted sender hosts) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApoEAGdHCUuERHMG/2dsb2JhbADTIQKEOgQ X-IronPort-AV: E=Sophos;i="4.47,267,1257112800"; d="scan'208";a="211286051" X-Authentication-Warning: fermat.math.technion.ac.il: nyh set sender to nyh@math.technion.ac.il using -f Date: Mon, 23 Nov 2009 00:16:55 +0200 From: "Nadav Har'El" To: Kazuo Teramoto Cc: zsh-users@zsh.org Subject: Re: Multi-core loops Message-ID: <20091122221655.GA28699@fermat.math.technion.ac.il> References: <20090628103129.GA15270@fermat.math.technion.ac.il> <5992e17f0911221305j53522ed6j55d8b1308fda2214@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5992e17f0911221305j53522ed6j55d8b1308fda2214@mail.gmail.com> User-Agent: Mutt/1.4.2.2i Hebrew-Date: 6 Kislev 5770 On Sun, Nov 22, 2009, Kazuo Teramoto wrote about "Re: Multi-core loops": > On Sun, Jun 28, 2009 at 8:31 AM, Nadav Har'El wrote: > > Zsh, like all shells, lets you easily do something many times in a loop. >... > > But when "dosomething" is CPU intensive, this is *not* what you'd want to > > do on a multi-core (multi CPU) machine, which have more-or-less become > > standard nowadays... > > Such a loop would only use one of the CPUs, and leave the other(s) unused. > > Instead, you'll want to keep all CPUs busy all the time, running M (=number > > of CPUs) processes at the same time. > > Any update on this? > > I'm searching for a solution. Perhaps this can't be done as a built-in > syntax but what about a more complex solution. I'm a noob (and with > the numbers of features of zsh, I'm gonna be a noob forever), and cant > find a small, beautiful, zsh-is-so-cool-ish solution for it and dont > know how to effective implement it e.g., without using python, I like > some zsh only solution. Unfortunately, no. I am running into this need very often - be it a small script to convert a bunch of media files, or a script to download a bunch of web pages, and so on, and always need to come up with some ugly half-working solution. I am still really surprised that no shell (that I know) comes with a convenient built-in syntax to do such loops. At one point I decided to go ahead an modify zsh myself. After some deliberation with myself, I came up with the following syntax: for i in 1 2 3 parallel 2 do echo $i sleep 3 done I.e., one adds the keyword "parallel" and the number of concurrent processes just before the "do/done" block. This syntax makes it easy to parrelize all kinds of for loops (C-like, csh-like, bourne-shell-like, etc.) in one syntax. What this should have done is to run the do/done block in the background (like with a &), and additionally block while 2 of these are already running, waiting until one of them stopped (we know when one stops because the shell has a SIGCHLD interrupt handler). Unfortunately, I found understanding the zsh parser much harder than I had originally anticipated. I managed to add the "parallel" syntax, but was not able (in the several hours I investing in trying to understand) how to generate the correct "instructions" (of the zsh virtual machine) to put the do/done block in the background, for example. All the tricks I tried with WCB_LIST, WCB_SUBLIST, WCB_PIPE, set_list_code, WCB_END and other strange things I tried, stopped short of actually working. And even if I had managed to pull that off, I still had some more missing pieces, like keeping a list of process ids of these backgrounded processes, recognizing (in the interrupt handler) when they're gone, and waiting until one of them is gone. > How people solved this problem any home-made solution for this, any tip? By the way, in my original thread I mentioned a post from 10 years ago (!) which suggested an eleborate trick to do what I was after: http://www.zsh.org/mla/users/1999/msg00644.html Have you tried this? -- Nadav Har'El | Sunday, Nov 22 2009, 6 Kislev 5770 nyh@math.technion.ac.il |----------------------------------------- Phone +972-523-790466, ICQ 13349191 |Seen on the back of a dump truck: http://nadav.harel.org.il |<---PASSING SIDE . . . . . SUICIDE--->