From: Uriel <uriel99@gmail.com>
To: Fans of the OS Plan 9 from Bell Labs <9fans@9fans.net>
Subject: Re: [9fans] threads vs forks
Date: Wed, 4 Mar 2009 14:23:14 +0100 [thread overview]
Message-ID: <5d375e920903040523j50625de0r9cdb6e3a51095d1b@mail.gmail.com> (raw)
In-Reply-To: <138575260903040333i262ec35cr345075427c2b6bed@mail.gmail.com>
What about xcpu?
On Wed, Mar 4, 2009 at 12:33 PM, hugo rivera <uair00@gmail.com> wrote:
> you are right. I was totally confused at the beggining.
> Thanks a lot.
>
> 2009/3/4, Vincent Schut <schut@sarvision.nl>:
>> hugo rivera wrote:
>>
>> > The cluster has torque installed as the resource manager. I think it
>> > runs of top of pbs (an older project).
>> > As far as I know now I just have to call a qsub command to submit my
>> > jobs on a queue, then the resource manager allocates a processor in
>> > the cluster for my process to run till is finished.
>> >
>>
>> Well, I don't know torque neither pbs, but I'm guessing that when you
>> submit a job, this job will be some program or script that is run on the
>> allocated processor? If so, your initial question of forking vs threading is
>> bogus. Your cluster manager will run (exec) your job, which if it is a
>> python script will start a python interpreter for each job. I guess that's
>> the overhead you get when running a flexible cluster system, flexible
>> meaning that it can run any type of job (shell script, binary executable,
>> python script, perl, etc.).
>> However, your overhead of starting new python processes each time may seem
>> significant when viewed in absolute terms, but if each job processes lots of
>> data and takes, as you said, 5 min to run on a decent processor, don't you
>> think the startup time for the python process would become non-significant?
>> For example, on a decent machine here, the first time python takes 0.224
>> secs to start and shutdown immediately, and consequetive starts take only
>> about 0.009 secs because everything is still in memory. Let's take the 0.224
>> secs for a worst case scenario. That would be approx 0.075 percent of your
>> job execution time. Now lets say you have 6 machines with 8 cores each and
>> perfect scaling, all your jobs would take 6000 / (6*8) *5min = 625 minutes
>> (10 hours 25 mins) without python starting each time, and 625 minutes and 28
>> seconds with python starting anew each job. Don't you think you could just
>> live with these 28 seconds more? Just reading this message might already
>> have taken you more than those 28 seconds...
>>
>> Vincent.
>>
>>
>>
>> > And I am not really sure if I have access to all the nodes, so I can
>> > install pp on each one of them.
>> >
>> > 2009/3/4, Vincent Schut <schut@sarvision.nl>:
>> >
>> > > hugo rivera wrote:
>> > >
>> > >
>> > > > Thanks for the advice.
>> > > > Nevertheless I am in no position to decide what pieces of software the
>> > > > cluster will run, I just have to deal with what I have, but anyway I
>> > > > can suggest other possibilities.
>> > > >
>> > > >
>> > > Well, depends on how you define 'software the cluster will run'. Do you
>> > > mean cluster management software, or really any program or script or
>> python
>> > > module that needs to be installed on each node? Because for pp, you
>> won't
>> > > need any cluster software. pp is just some python module and helper
>> scripts.
>> > > You *do* need to install this (pure python) module on each node, yes,
>> but
>> > > that's it, nothing else needed.
>> > > Btw, you said 'it's a small cluster, about 6 machines'. Now I'm not an
>> > > expert, but I don't think you can do threading/forking from one machine
>> to
>> > > another (on linux). So I suppose there already is some cluster
>> management
>> > > software involved? And while you appear to be "in no position to decide
>> what
>> > > pieces of software the cluster will run", you might want to enlighten us
>> on
>> > > what this cluster /will/ run? Your best solution might depend on that...
>> > >
>> > > Cheers,
>> > > Vincent.
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>>
>>
>>
>
>
> --
> Hugo
>
>
next prev parent reply other threads:[~2009-03-04 13:23 UTC|newest]
Thread overview: 71+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-03-03 11:52 hugo rivera
2009-03-03 15:19 ` David Leimbach
2009-03-03 15:32 ` Uriel
2009-03-03 16:15 ` hugo rivera
2009-03-03 15:33 ` hugo rivera
2009-03-03 18:11 ` Roman V. Shaposhnik
2009-03-03 18:38 ` Bakul Shah
2009-03-06 18:47 ` Roman V Shaposhnik
2009-03-06 20:38 ` David Leimbach
2009-03-07 8:00 ` Bakul Shah
2009-03-07 0:21 ` Bakul Shah
2009-03-07 2:20 ` Brian L. Stuart
2009-03-03 23:08 ` J.R. Mauro
2009-03-03 23:15 ` Uriel
2009-03-03 23:23 ` J.R. Mauro
2009-03-03 23:54 ` Devon H. O'Dell
2009-03-04 0:33 ` J.R. Mauro
2009-03-04 0:54 ` erik quanstrom
2009-03-04 1:54 ` J.R. Mauro
2009-03-04 3:18 ` James Tomaschke
2009-03-04 3:30 ` erik quanstrom
2009-03-04 4:44 ` James Tomaschke
2009-03-04 5:05 ` J.R. Mauro
2009-03-04 5:50 ` erik quanstrom
2009-03-04 6:08 ` andrey mirtchovski
2009-03-04 16:52 ` J.R. Mauro
2009-03-04 17:14 ` ron minnich
2009-03-04 17:27 ` William Josephson
2009-03-04 18:15 ` erik quanstrom
2009-03-05 3:32 ` J.R. Mauro
2009-03-05 3:39 ` erik quanstrom
2009-03-05 3:55 ` William K. Josephson
2009-03-05 4:00 ` erik quanstrom
2009-03-05 4:16 ` William K. Josephson
2009-03-07 3:01 ` William Josephson
2009-03-07 3:31 ` erik quanstrom
2009-03-07 6:00 ` William Josephson
2009-03-07 13:58 ` erik quanstrom
2009-03-07 14:37 ` William Josephson
2009-03-07 15:05 ` erik quanstrom
2009-03-07 15:28 ` William K. Josephson
2009-03-07 5:00 ` lucio
2009-03-07 5:08 ` William Josephson
2009-03-07 5:19 ` erik quanstrom
2009-03-07 5:45 ` [9fans] Flash William K. Josephson
2009-03-07 14:42 ` erik quanstrom
2009-03-07 14:56 ` William Josephson
2009-03-07 15:39 ` Russ Cox
2009-03-07 16:34 ` erik quanstrom
2009-03-07 5:24 ` [9fans] threads vs forks lucio
2009-03-04 5:19 ` David Leimbach
2009-03-04 2:47 ` John Barham
2009-03-04 5:24 ` blstuart
2009-03-04 5:37 ` erik quanstrom
2009-03-04 16:29 ` Roman V Shaposhnik
2009-03-04 16:56 ` john
2009-03-06 9:39 ` maht
2009-03-04 5:07 ` David Leimbach
2009-03-04 5:35 ` John Barham
2009-03-03 16:00 ` ron minnich
2009-03-03 16:28 ` hugo rivera
2009-03-03 17:31 ` ron minnich
2009-03-03 16:47 ` John Barham
2009-03-04 9:37 ` Vincent Schut
2009-03-04 9:58 ` hugo rivera
2009-03-04 10:30 ` Vincent Schut
2009-03-04 10:45 ` hugo rivera
2009-03-04 11:15 ` Vincent Schut
2009-03-04 11:33 ` hugo rivera
2009-03-04 13:23 ` Uriel [this message]
2009-03-04 14:57 ` ron minnich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5d375e920903040523j50625de0r9cdb6e3a51095d1b@mail.gmail.com \
--to=uriel99@gmail.com \
--cc=9fans@9fans.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).