From: John Barham <jbarham@gmail.com>
To: Fans of the OS Plan 9 from Bell Labs <9fans@9fans.net>
Subject: Re: [9fans] threads vs forks
Date: Tue, 3 Mar 2009 08:47:25 -0800 [thread overview]
Message-ID: <4f34febc0903030847t9aedad9haf4355e74953e6a3@mail.gmail.com> (raw)
In-Reply-To: <138575260903030352s623807d7p5a3075b1f7a591f6@mail.gmail.com>
On Tue, Mar 3, 2009 at 3:52 AM, hugo rivera <uair00@gmail.com> wrote:
> I have to launch many tasks running in parallel (~5000) in a
> cluster running linux. Each of the task performs some astronomical
> calculations and I am not pretty sure if using fork is the best answer
> here.
> First of all, all the programming is done in python and c...
Take a look at the multiprocessing package
(http://docs.python.org/library/multiprocessing.html), newly
introduced with Python 2.6 and 3.0:
"multiprocessing is a package that supports spawning processes using
an API similar to the threading module. The multiprocessing package
offers both local and remote concurrency, effectively side-stepping
the Global Interpreter Lock by using subprocesses instead of threads."
It should be a quick and easy way to set up a cluster-wide job
processing system (provided all your jobs are driven by Python).
It also looks like it's been (partially?) back-ported to Python 2.4
and 2.5: http://pypi.python.org/pypi/processing.
John
next prev parent reply other threads:[~2009-03-03 16:47 UTC|newest]
Thread overview: 71+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-03-03 11:52 hugo rivera
2009-03-03 15:19 ` David Leimbach
2009-03-03 15:32 ` Uriel
2009-03-03 16:15 ` hugo rivera
2009-03-03 15:33 ` hugo rivera
2009-03-03 18:11 ` Roman V. Shaposhnik
2009-03-03 18:38 ` Bakul Shah
2009-03-06 18:47 ` Roman V Shaposhnik
2009-03-06 20:38 ` David Leimbach
2009-03-07 8:00 ` Bakul Shah
2009-03-07 0:21 ` Bakul Shah
2009-03-07 2:20 ` Brian L. Stuart
2009-03-03 23:08 ` J.R. Mauro
2009-03-03 23:15 ` Uriel
2009-03-03 23:23 ` J.R. Mauro
2009-03-03 23:54 ` Devon H. O'Dell
2009-03-04 0:33 ` J.R. Mauro
2009-03-04 0:54 ` erik quanstrom
2009-03-04 1:54 ` J.R. Mauro
2009-03-04 3:18 ` James Tomaschke
2009-03-04 3:30 ` erik quanstrom
2009-03-04 4:44 ` James Tomaschke
2009-03-04 5:05 ` J.R. Mauro
2009-03-04 5:50 ` erik quanstrom
2009-03-04 6:08 ` andrey mirtchovski
2009-03-04 16:52 ` J.R. Mauro
2009-03-04 17:14 ` ron minnich
2009-03-04 17:27 ` William Josephson
2009-03-04 18:15 ` erik quanstrom
2009-03-05 3:32 ` J.R. Mauro
2009-03-05 3:39 ` erik quanstrom
2009-03-05 3:55 ` William K. Josephson
2009-03-05 4:00 ` erik quanstrom
2009-03-05 4:16 ` William K. Josephson
2009-03-07 3:01 ` William Josephson
2009-03-07 3:31 ` erik quanstrom
2009-03-07 6:00 ` William Josephson
2009-03-07 13:58 ` erik quanstrom
2009-03-07 14:37 ` William Josephson
2009-03-07 15:05 ` erik quanstrom
2009-03-07 15:28 ` William K. Josephson
2009-03-07 5:00 ` lucio
2009-03-07 5:08 ` William Josephson
2009-03-07 5:19 ` erik quanstrom
2009-03-07 5:45 ` [9fans] Flash William K. Josephson
2009-03-07 14:42 ` erik quanstrom
2009-03-07 14:56 ` William Josephson
2009-03-07 15:39 ` Russ Cox
2009-03-07 16:34 ` erik quanstrom
2009-03-07 5:24 ` [9fans] threads vs forks lucio
2009-03-04 5:19 ` David Leimbach
2009-03-04 2:47 ` John Barham
2009-03-04 5:24 ` blstuart
2009-03-04 5:37 ` erik quanstrom
2009-03-04 16:29 ` Roman V Shaposhnik
2009-03-04 16:56 ` john
2009-03-06 9:39 ` maht
2009-03-04 5:07 ` David Leimbach
2009-03-04 5:35 ` John Barham
2009-03-03 16:00 ` ron minnich
2009-03-03 16:28 ` hugo rivera
2009-03-03 17:31 ` ron minnich
2009-03-03 16:47 ` John Barham [this message]
2009-03-04 9:37 ` Vincent Schut
2009-03-04 9:58 ` hugo rivera
2009-03-04 10:30 ` Vincent Schut
2009-03-04 10:45 ` hugo rivera
2009-03-04 11:15 ` Vincent Schut
2009-03-04 11:33 ` hugo rivera
2009-03-04 13:23 ` Uriel
2009-03-04 14:57 ` ron minnich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4f34febc0903030847t9aedad9haf4355e74953e6a3@mail.gmail.com \
--to=jbarham@gmail.com \
--cc=9fans@9fans.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).