Re: [9fans] xcpu note

[Ronald G Minnich <rminnich@lanl.gov>]

David Leimbach wrote:

> Clustermatic is pretty cool, I think it's what was installed on one of
> the other clusters I used at LANL as a contractor at the time.  I
> recall a companion tool for bproc to request nodes, sort of an ad-hoc
> scheduler.  I had to integrate support for this in our MPI's start up
> that I was testing on that machine.

the simple scheduler, bjs, was written by erik hendriks (now at Google, 
sigh) and was rock-solid. It ran on one cluster, unattended, scheduling 
128 2-cpu nodes with a very diverse job mix, for one year. It was a 
great piece of software. It was far faster, and far more reliable, than 
any scheduler we have ever seen, then or now. In one test, we ran about 
20,000 jobs through it on about an hour, on a 1024-node cluster, just to 
test. Note that it could probably have scheduled a lot more jobs, but 
the run-time of the job was non-zero. No other scheduler we have used 
comes close to this kind of performance. Scheduler overhead was 
basically insignificant.

> 
> I'm curious to see how this all fits together with xcpu, if there is
> such a resource allocation setup needed etc.

we're going to take bjs and have it schedule nodes to give to users.

Note one thing we are going to do with xcpu: attach nodes to a user's 
desktop machine, rather than make users log in to the cluster. So users 
will get interactive clusters that look like they own them. This will, 
we hope, kill batch mode. Plan 9 ideas make this possible. It's going to 
be a big change, one we hope users will like.

If you look at how most clusters are used today, they closely resemble 
the batch world of the 1960s. It is actually kind of shocking. I 
downloaded a JCL manual a year or two ago, and compared what JCL did to 
what people wanted batch schedulers for clusters to do, and the 
correspondance was a little depressing. The Data General ad said it 
best: "Batch is a bitch".

Oh yeah, if anyone has a copy of that ad (Google does not), i'd like it 
in .pdf :-) It appeared in the late 70s IIRC.

ron
p.s. go ahead, google JCL, and you can find very recent manuals on how 
to use it. I will be happy to post the JCL for "sort + copy" if anyone 
wants to see it.