From: hugo rivera <uair00@gmail.com>
To: Fans of the OS Plan 9 from Bell Labs <9fans@9fans.net>
Subject: Re: [9fans] threads vs forks
Date: Wed, 4 Mar 2009 12:33:59 +0100 [thread overview]
Message-ID: <138575260903040333i262ec35cr345075427c2b6bed@mail.gmail.com> (raw)
In-Reply-To: <golnt7$hkj$1@ger.gmane.org>
you are right. I was totally confused at the beggining.
Thanks a lot.
2009/3/4, Vincent Schut <schut@sarvision.nl>:
> hugo rivera wrote:
>
> > The cluster has torque installed as the resource manager. I think it
> > runs of top of pbs (an older project).
> > As far as I know now I just have to call a qsub command to submit my
> > jobs on a queue, then the resource manager allocates a processor in
> > the cluster for my process to run till is finished.
> >
>
> Well, I don't know torque neither pbs, but I'm guessing that when you
> submit a job, this job will be some program or script that is run on the
> allocated processor? If so, your initial question of forking vs threading is
> bogus. Your cluster manager will run (exec) your job, which if it is a
> python script will start a python interpreter for each job. I guess that's
> the overhead you get when running a flexible cluster system, flexible
> meaning that it can run any type of job (shell script, binary executable,
> python script, perl, etc.).
> However, your overhead of starting new python processes each time may seem
> significant when viewed in absolute terms, but if each job processes lots of
> data and takes, as you said, 5 min to run on a decent processor, don't you
> think the startup time for the python process would become non-significant?
> For example, on a decent machine here, the first time python takes 0.224
> secs to start and shutdown immediately, and consequetive starts take only
> about 0.009 secs because everything is still in memory. Let's take the 0.224
> secs for a worst case scenario. That would be approx 0.075 percent of your
> job execution time. Now lets say you have 6 machines with 8 cores each and
> perfect scaling, all your jobs would take 6000 / (6*8) *5min = 625 minutes
> (10 hours 25 mins) without python starting each time, and 625 minutes and 28
> seconds with python starting anew each job. Don't you think you could just
> live with these 28 seconds more? Just reading this message might already
> have taken you more than those 28 seconds...
>
> Vincent.
>
>
>
> > And I am not really sure if I have access to all the nodes, so I can
> > install pp on each one of them.
> >
> > 2009/3/4, Vincent Schut <schut@sarvision.nl>:
> >
> > > hugo rivera wrote:
> > >
> > >
> > > > Thanks for the advice.
> > > > Nevertheless I am in no position to decide what pieces of software the
> > > > cluster will run, I just have to deal with what I have, but anyway I
> > > > can suggest other possibilities.
> > > >
> > > >
> > > Well, depends on how you define 'software the cluster will run'. Do you
> > > mean cluster management software, or really any program or script or
> python
> > > module that needs to be installed on each node? Because for pp, you
> won't
> > > need any cluster software. pp is just some python module and helper
> scripts.
> > > You *do* need to install this (pure python) module on each node, yes,
> but
> > > that's it, nothing else needed.
> > > Btw, you said 'it's a small cluster, about 6 machines'. Now I'm not an
> > > expert, but I don't think you can do threading/forking from one machine
> to
> > > another (on linux). So I suppose there already is some cluster
> management
> > > software involved? And while you appear to be "in no position to decide
> what
> > > pieces of software the cluster will run", you might want to enlighten us
> on
> > > what this cluster /will/ run? Your best solution might depend on that...
> > >
> > > Cheers,
> > > Vincent.
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
--
Hugo
next prev parent reply other threads:[~2009-03-04 11:33 UTC|newest]
Thread overview: 71+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-03-03 11:52 hugo rivera
2009-03-03 15:19 ` David Leimbach
2009-03-03 15:32 ` Uriel
2009-03-03 16:15 ` hugo rivera
2009-03-03 15:33 ` hugo rivera
2009-03-03 18:11 ` Roman V. Shaposhnik
2009-03-03 18:38 ` Bakul Shah
2009-03-06 18:47 ` Roman V Shaposhnik
2009-03-06 20:38 ` David Leimbach
2009-03-07 8:00 ` Bakul Shah
2009-03-07 0:21 ` Bakul Shah
2009-03-07 2:20 ` Brian L. Stuart
2009-03-03 23:08 ` J.R. Mauro
2009-03-03 23:15 ` Uriel
2009-03-03 23:23 ` J.R. Mauro
2009-03-03 23:54 ` Devon H. O'Dell
2009-03-04 0:33 ` J.R. Mauro
2009-03-04 0:54 ` erik quanstrom
2009-03-04 1:54 ` J.R. Mauro
2009-03-04 3:18 ` James Tomaschke
2009-03-04 3:30 ` erik quanstrom
2009-03-04 4:44 ` James Tomaschke
2009-03-04 5:05 ` J.R. Mauro
2009-03-04 5:50 ` erik quanstrom
2009-03-04 6:08 ` andrey mirtchovski
2009-03-04 16:52 ` J.R. Mauro
2009-03-04 17:14 ` ron minnich
2009-03-04 17:27 ` William Josephson
2009-03-04 18:15 ` erik quanstrom
2009-03-05 3:32 ` J.R. Mauro
2009-03-05 3:39 ` erik quanstrom
2009-03-05 3:55 ` William K. Josephson
2009-03-05 4:00 ` erik quanstrom
2009-03-05 4:16 ` William K. Josephson
2009-03-07 3:01 ` William Josephson
2009-03-07 3:31 ` erik quanstrom
2009-03-07 6:00 ` William Josephson
2009-03-07 13:58 ` erik quanstrom
2009-03-07 14:37 ` William Josephson
2009-03-07 15:05 ` erik quanstrom
2009-03-07 15:28 ` William K. Josephson
2009-03-07 5:00 ` lucio
2009-03-07 5:08 ` William Josephson
2009-03-07 5:19 ` erik quanstrom
2009-03-07 5:45 ` [9fans] Flash William K. Josephson
2009-03-07 14:42 ` erik quanstrom
2009-03-07 14:56 ` William Josephson
2009-03-07 15:39 ` Russ Cox
2009-03-07 16:34 ` erik quanstrom
2009-03-07 5:24 ` [9fans] threads vs forks lucio
2009-03-04 5:19 ` David Leimbach
2009-03-04 2:47 ` John Barham
2009-03-04 5:24 ` blstuart
2009-03-04 5:37 ` erik quanstrom
2009-03-04 16:29 ` Roman V Shaposhnik
2009-03-04 16:56 ` john
2009-03-06 9:39 ` maht
2009-03-04 5:07 ` David Leimbach
2009-03-04 5:35 ` John Barham
2009-03-03 16:00 ` ron minnich
2009-03-03 16:28 ` hugo rivera
2009-03-03 17:31 ` ron minnich
2009-03-03 16:47 ` John Barham
2009-03-04 9:37 ` Vincent Schut
2009-03-04 9:58 ` hugo rivera
2009-03-04 10:30 ` Vincent Schut
2009-03-04 10:45 ` hugo rivera
2009-03-04 11:15 ` Vincent Schut
2009-03-04 11:33 ` hugo rivera [this message]
2009-03-04 13:23 ` Uriel
2009-03-04 14:57 ` ron minnich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=138575260903040333i262ec35cr345075427c2b6bed@mail.gmail.com \
--to=uair00@gmail.com \
--cc=9fans@9fans.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).