From mboxrd@z Thu Jan 1 00:00:00 1970 MIME-Version: 1.0 In-Reply-To: <138575260903030352s623807d7p5a3075b1f7a591f6@mail.gmail.com> References: <138575260903030352s623807d7p5a3075b1f7a591f6@mail.gmail.com> Date: Tue, 3 Mar 2009 08:47:25 -0800 Message-ID: <4f34febc0903030847t9aedad9haf4355e74953e6a3@mail.gmail.com> From: John Barham To: Fans of the OS Plan 9 from Bell Labs <9fans@9fans.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [9fans] threads vs forks Topicbox-Message-UUID: ace9f884-ead4-11e9-9d60-3106f5b1d025 On Tue, Mar 3, 2009 at 3:52 AM, hugo rivera wrote: > I have to launch many tasks running in parallel (~5000) in a > cluster running linux. Each of the task performs some astronomical > calculations and I am not pretty sure if using fork is the best answer > here. > First of all, all the programming is done in python and c... Take a look at the multiprocessing package (http://docs.python.org/library/multiprocessing.html), newly introduced with Python 2.6 and 3.0: "multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads." It should be a quick and easy way to set up a cluster-wide job processing system (provided all your jobs are driven by Python). It also looks like it's been (partially?) back-ported to Python 2.4 and 2.5: http://pypi.python.org/pypi/processing. John