[9fans] Distributed Pipelines

[9fans] Distributed Pipelines

[Akshat Kumar]

In running some computationally intensive processes
between my terminal(s) and cpu server(s), I noticed
a problem in trying to join data together by a pipeline:

procdata | cpu -c 'process'

doesn't really send the output of `procdata' to `process',
as the latter is being run on a CPU server. Since there
are some use cases where `procdata' might take a long
time to get to EOF (or possibly may never reach it), it
isn't in my interest to first send the output of `procdata'
to a file and then run `process' on the CPU after it's
finished.
Likewise, there is the inverse problem:

cpu -c 'procdata' | process

outputs to rio terminal instead of sending output to `process'.
It seems to me that the two problems would have similar
solutions.

Perhaps I'm overlooking some simple solutions here.
Any suggestions?


Best,
ak

Re: [9fans] Distributed Pipelines

[Steve Simon]

> cpu -c 'procdata' | process
> ...
> Perhaps I'm overlooking some simple solutions here.
> Any suggestions?

cpu(1) works by starting exportfs on the remote machine and serving
the local machines filespace. The remote shell is started with its
stdin/out/err attached to /mnt/term/dev/cons, thus the command you
tried will not work (by design).

what you want is rx(1) which does exactly what you want, somthing
like rsh(1) from the Unix world, except it uses plan9' secure
authentication; e.g.:

	rx $cpu | process

-Steve

Re: [9fans] Distributed Pipelines

[Akshat Kumar]

Thanks Steve,

rx $cpu 'procdata' | process

works well for one way.
However,

procdata | rx $cpu 'process'

is in the same way as with cpu(1).
Any suggestions for piping in that
direction?


Best,
ak

On Sun, Apr 25, 2010 at 5:40 PM, Steve Simon <steve@quintile.net> wrote:
>> cpu -c 'procdata' | process
>> ...
>> Perhaps I'm overlooking some simple solutions here.
>> Any suggestions?
>
> cpu(1) works by starting exportfs on the remote machine and serving
> the local machines filespace. The remote shell is started with its
> stdin/out/err attached to /mnt/term/dev/cons, thus the command you
> tried will not work (by design).
>
> what you want is rx(1) which does exactly what you want, somthing
> like rsh(1) from the Unix world, except it uses plan9' secure
> authentication; e.g.:
>
>        rx $cpu | process
>
> -Steve
>
>

Re: [9fans] Distributed Pipelines

[Eric Van Hensbergen]

Take a look at Noah's PUSH shell.  It's not there yet, but maybe later
today.

Sent from my iPhone

On Apr 26, 2010, at 2:50 AM, Akshat Kumar
<akumar@mail.nanosouffle.net> wrote:

> Thanks Steve,
>
> rx $cpu 'procdata' | process
>
> works well for one way.
> However,
>
> procdata | rx $cpu 'process'
>
> is in the same way as with cpu(1).
> Any suggestions for piping in that
> direction?
>
>
> Best,
> ak
>
> On Sun, Apr 25, 2010 at 5:40 PM, Steve Simon <steve@quintile.net>
> wrote:
>>> cpu -c 'procdata' | process
>>> ...
>>> Perhaps I'm overlooking some simple solutions here.
>>> Any suggestions?
>>
>> cpu(1) works by starting exportfs on the remote machine and serving
>> the local machines filespace. The remote shell is started with its
>> stdin/out/err attached to /mnt/term/dev/cons, thus the command you
>> tried will not work (by design).
>>
>> what you want is rx(1) which does exactly what you want, somthing
>> like rsh(1) from the Unix world, except it uses plan9' secure
>> authentication; e.g.:
>>
>>        rx $cpu | process
>>
>> -Steve
>>
>>
>

Re: [9fans] Distributed Pipelines

[Akshat Kumar]

Hi Eric,

The only reference to PUSH I see is
at http://code.google.com/p/push
where the site reads,

"This is the new unix port of push."

Where might I find the native Plan 9
version?


Best,
ak


On 4/25/10, Eric Van Hensbergen <ericvh@gmail.com> wrote:
> Take a look at Noah's PUSH shell.  It's not there yet, but maybe later
> today.
>
> Sent from my iPhone
>
> On Apr 26, 2010, at 2:50 AM, Akshat Kumar
> <akumar@mail.nanosouffle.net> wrote:
>
>> Thanks Steve,
>>
>> rx $cpu 'procdata' | process
>>
>> works well for one way.
>> However,
>>
>> procdata | rx $cpu 'process'
>>
>> is in the same way as with cpu(1).
>> Any suggestions for piping in that
>> direction?
>>
>>
>> Best,
>> ak
>>
>> On Sun, Apr 25, 2010 at 5:40 PM, Steve Simon <steve@quintile.net>
>> wrote:
>>>> cpu -c 'procdata' | process
>>>> ...
>>>> Perhaps I'm overlooking some simple solutions here.
>>>> Any suggestions?
>>>
>>> cpu(1) works by starting exportfs on the remote machine and serving
>>> the local machines filespace. The remote shell is started with its
>>> stdin/out/err attached to /mnt/term/dev/cons, thus the command you
>>> tried will not work (by design).
>>>
>>> what you want is rx(1) which does exactly what you want, somthing
>>> like rsh(1) from the Unix world, except it uses plan9' secure
>>> authentication; e.g.:
>>>
>>>        rx $cpu | process
>>>
>>> -Steve
>>>
>>>
>>
>
>

Re: [9fans] Distributed Pipelines

[Eric Van Hensbergen]

The version there is Plan9ports and should work under Plan 9 as well -- if it doesn't, beat on Noah :)

     -eric

On Apr 26, 2010, at 9:33 AM, Akshat Kumar wrote:

> Hi Eric,
> 
> The only reference to PUSH I see is
> at http://code.google.com/p/push
> where the site reads,
> 
> "This is the new unix port of push."
> 
> Where might I find the native Plan 9
> version?
> 
> 
> Best,
> ak
> 
> 
> On 4/25/10, Eric Van Hensbergen <ericvh@gmail.com> wrote:
>> Take a look at Noah's PUSH shell.  It's not there yet, but maybe later
>> today.
>> 
>> Sent from my iPhone
>> 
>> On Apr 26, 2010, at 2:50 AM, Akshat Kumar
>> <akumar@mail.nanosouffle.net> wrote:
>> 
>>> Thanks Steve,
>>> 
>>> rx $cpu 'procdata' | process
>>> 
>>> works well for one way.
>>> However,
>>> 
>>> procdata | rx $cpu 'process'
>>> 
>>> is in the same way as with cpu(1).
>>> Any suggestions for piping in that
>>> direction?
>>> 
>>> 
>>> Best,
>>> ak
>>> 
>>> On Sun, Apr 25, 2010 at 5:40 PM, Steve Simon <steve@quintile.net>
>>> wrote:
>>>>> cpu -c 'procdata' | process
>>>>> ...
>>>>> Perhaps I'm overlooking some simple solutions here.
>>>>> Any suggestions?
>>>> 
>>>> cpu(1) works by starting exportfs on the remote machine and serving
>>>> the local machines filespace. The remote shell is started with its
>>>> stdin/out/err attached to /mnt/term/dev/cons, thus the command you
>>>> tried will not work (by design).
>>>> 
>>>> what you want is rx(1) which does exactly what you want, somthing
>>>> like rsh(1) from the Unix world, except it uses plan9' secure
>>>> authentication; e.g.:
>>>> 
>>>>       rx $cpu | process
>>>> 
>>>> -Steve
>>>> 
>>>> 
>>> 
>> 
>> 
> 

Re: [9fans] Distributed Pipelines

[Akshat Kumar]

For the record: rx(1) man pages imply
the sort of behaviour from rx(1) that I
would like:

"...
          eqn paper | rx kremvax troff -ms | rx deepthought lp
               Parallel processing: do each stage of a pipeline on a
               different machine.
"

however, it seems not to work this way.
My basic test has been something like:

echo '1 2 3' | rx $cpu awk -f $home/comp.awk | gview

... just a simple sample use case.
Is there something special about awk(1)
that would cause it not to receive
standard input the way rx(1) provides it?
Even just:

rx $cpu awk -f $home/comp.awk

wants to take standard input, but
apparently doesn't get it.


Thanks,
ak


On 4/26/10, Eric Van Hensbergen <ericvh@gmail.com> wrote:
> The version there is Plan9ports and should work under Plan 9 as well -- if
> it doesn't, beat on Noah :)
>
>      -eric
>
> On Apr 26, 2010, at 9:33 AM, Akshat Kumar wrote:
>
>> Hi Eric,
>>
>> The only reference to PUSH I see is
>> at http://code.google.com/p/push
>> where the site reads,
>>
>> "This is the new unix port of push."
>>
>> Where might I find the native Plan 9
>> version?
>>
>>
>> Best,
>> ak
>>
>>
>> On 4/25/10, Eric Van Hensbergen <ericvh@gmail.com> wrote:
>>> Take a look at Noah's PUSH shell.  It's not there yet, but maybe later
>>> today.
>>>
>>> Sent from my iPhone
>>>
>>> On Apr 26, 2010, at 2:50 AM, Akshat Kumar
>>> <akumar@mail.nanosouffle.net> wrote:
>>>
>>>> Thanks Steve,
>>>>
>>>> rx $cpu 'procdata' | process
>>>>
>>>> works well for one way.
>>>> However,
>>>>
>>>> procdata | rx $cpu 'process'
>>>>
>>>> is in the same way as with cpu(1).
>>>> Any suggestions for piping in that
>>>> direction?
>>>>
>>>>
>>>> Best,
>>>> ak
>>>>
>>>> On Sun, Apr 25, 2010 at 5:40 PM, Steve Simon <steve@quintile.net>
>>>> wrote:
>>>>>> cpu -c 'procdata' | process
>>>>>> ...
>>>>>> Perhaps I'm overlooking some simple solutions here.
>>>>>> Any suggestions?
>>>>>
>>>>> cpu(1) works by starting exportfs on the remote machine and serving
>>>>> the local machines filespace. The remote shell is started with its
>>>>> stdin/out/err attached to /mnt/term/dev/cons, thus the command you
>>>>> tried will not work (by design).
>>>>>
>>>>> what you want is rx(1) which does exactly what you want, somthing
>>>>> like rsh(1) from the Unix world, except it uses plan9' secure
>>>>> authentication; e.g.:
>>>>>
>>>>>       rx $cpu | process
>>>>>
>>>>> -Steve
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
>

Re: [9fans] Distributed Pipelines

[erik quanstrom]

>
> "...
>           eqn paper | rx kremvax troff -ms | rx deepthought lp
>                Parallel processing: do each stage of a pipeline on a
>                different machine.
> "
>
> however, it seems not to work this way.
> My basic test has been something like:
>
> echo '1 2 3' | rx $cpu awk -f $home/comp.awk | gview

cool that you tracked this down.

this just doesn't work.  the problem seems to be that tcp
is eating the eof.  awk doesn't know to exit.

if you are using il, this does work.

	; echo 1 2 3| 8.rx bureau sed s/1/x/ | sed s/2/y/
	x y 3

but if you are using tcp, it hangs.  sed never sees the
eof and doesn't generate output (because it's buffered).

the answer isn't obvious to me with tcp as i don't
know of a way to half-close a tcp connection —
from userspace anyway.

- erik

Re: [9fans] Distributed Pipelines

[Akshat Kumar]

Hi Erik,

Thanks for figuring that bit out!
Indeed, it seems TCP is the
problem, and IL seems to work
fine for me for the moment:

echo '1 2 3' | rx il!$cpu!17009 awk -f $home/comp.awk | gview

works perfectly!

I'll try to dig deeper into the TCP case.


Best,
ak


On 4/26/10, erik quanstrom <quanstro@labs.coraid.com> wrote:
>>
>> "...
>>           eqn paper | rx kremvax troff -ms | rx deepthought lp
>>                Parallel processing: do each stage of a pipeline on a
>>                different machine.
>> "
>>
>> however, it seems not to work this way.
>> My basic test has been something like:
>>
>> echo '1 2 3' | rx $cpu awk -f $home/comp.awk | gview
>
> cool that you tracked this down.
>
> this just doesn't work.  the problem seems to be that tcp
> is eating the eof.  awk doesn't know to exit.
>
> if you are using il, this does work.
>
> 	; echo 1 2 3| 8.rx bureau sed s/1/x/ | sed s/2/y/
> 	x y 3
>
> but if you are using tcp, it hangs.  sed never sees the
> eof and doesn't generate output (because it's buffered).
>
> the answer isn't obvious to me with tcp as i don't
> know of a way to half-close a tcp connection —
> from userspace anyway.
>
> - erik
>
>

Re: [9fans] Distributed Pipelines

[erik quanstrom]

On Mon Apr 26 18:04:40 EDT 2010, akumar@mail.nanosouffle.net wrote:
> Hi Erik,
>
> Thanks for figuring that bit out!
> Indeed, it seems TCP is the
> problem, and IL seems to work
> fine for me for the moment:
>
> echo '1 2 3' | rx il!$cpu!17009 awk -f $home/comp.awk | gview
>
> works perfectly!
>
> I'll try to dig deeper into the TCP case.

tcp doesn't preserve message boundaries, so
the 0-byte write should get tossed.  i don't think
there's any hope with rx's current setup.  you'd
need to resort to stuffing or some other how-to-
hide-yer-oob data trick or alternately a tcp
half-close.

- erik

Re: [9fans] Distributed Pipelines

[Tim Newsham]

What about some mounting/binding hackery where you replace
/dev/cons so that the original "cpu" command works?

Tim Newsham | www.thenewsh.com/~newsham | thenewsh.blogspot.com

Re: [9fans] Distributed Pipelines

[lucio]

> What about some mounting/binding hackery where you replace
> /dev/cons so that the original "cpu" command works?

I was going to suggest using UDP instead of TCP or IL. Is that a silly
idea?

++L

Re: [9fans] Distributed Pipelines

[Derek Fawcus]

On Mon, Apr 26, 2010 at 06:17:38PM -0400, erik quanstrom wrote:
> you'd
> need to resort to stuffing or some other how-to-
> hide-yer-oob data trick or alternately a tcp
> half-close.

Urgent pointer?  but the half close sounds 'better'.

Re: [9fans] Distributed Pipelines

[erik quanstrom]

On Tue Apr 27 00:33:52 EDT 2010, lucio@proxima.alt.za wrote:
> > What about some mounting/binding hackery where you replace
> > /dev/cons so that the original "cpu" command works?
>
> I was going to suggest using UDP instead of TCP or IL. Is that a silly
> idea?

cpu/rx require a stream protocol.  udp will not do.

- erik

Re: [9fans] Distributed Pipelines

[erik quanstrom]

On Tue Apr 27 00:31:03 EDT 2010, newsham@lava.net wrote:
> What about some mounting/binding hackery where you replace
> /dev/cons so that the original "cpu" command works?

why the resistance to il?  rx is a good example of il's strengths.
in order for cpu to work, it uses 2 extra processes.  rx is much
more efficient.  (and 1/4 the code) great for your trusted network.
or perhaps your local supercomputer.

rx doesn't do encryption.  a srx using ssl/tls would be
able to sneak a 0 write through since the record layer
should expand that into a application record with
application data length of 0.

- erik

Re: [9fans] Distributed Pipelines

[Ryousei Takano]

On Tue, Apr 27, 2010 at 9:46 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> On Tue Apr 27 00:31:03 EDT 2010, newsham@lava.net wrote:
>> What about some mounting/binding hackery where you replace
>> /dev/cons so that the original "cpu" command works?
>
> why the resistance to il?  rx is a good example of il's strengths.
> in order for cpu to work, it uses 2 extra processes.  rx is much
> more efficient.  (and 1/4 the code) great for your trusted network.
> or perhaps your local supercomputer.
>
> rx doesn't do encryption.  a srx using ssl/tls would be
> able to sneak a 0 write through since the record layer
> should expand that into a application record with
> application data length of 0.
>
I became interested in IL protocol.
Please tell me why IL is removed from the main distribution.

Thanks,
Ryousei

Re: [9fans] Distributed Pipelines

[David du Colombier]

> Please tell me why IL is removed from the main distribution.

From the Fourth Edition Release Notes [1] :

"We are phasing out the IL protocol since it doesn't
handle long-distance connections well (and long-distance
networks don't handle it well, either)"

[1] http://plan9.bell-labs.com/sys/doc/release4.html

-- 
David du Colombier

Re: [9fans] Distributed Pipelines

[Ryousei Takano]

Hi David,

On Sunday, May 2, 2010, David du Colombier <0intro@gmail.com> wrote:
>> Please tell me why IL is removed from the main distribution.
>
> From the Fourth Edition Release Notes [1] :
>
> "We are phasing out the IL protocol since it doesn't
> handle long-distance connections well (and long-distance
> networks don't handle it well, either)"
>
> [1] http://plan9.bell-labs.com/sys/doc/release4.html
>

Thanks for your quick response.
Does it mean IL has performance issue on long-distance networks?

Thanks,
Ryousei

Re: [9fans] Distributed Pipelines

[David du Colombier]

> Does it mean IL has performance issue on long-distance networks?

As I understand it, the real problem is that Internet
doesn't handle IL well.

--
David du Colombier

Re: [9fans] Distributed Pipelines

[Russ Cox]

On Mon, May 3, 2010 at 11:10 AM, David du Colombier <0intro@gmail.com> wrote:
>> Does it mean IL has performance issue on long-distance networks?
>
> As I understand it, the real problem is that Internet
> doesn't handle IL well.

They are both problems.  Routing issues aside,
IL is particularly bad on asymmetric networks like
cable modems (download much faster than upload).

Russ

Re: [9fans] Distributed Pipelines

[Jorden M]

On Mon, May 3, 2010 at 4:22 PM, Russ Cox <rsc@swtch.com> wrote:
> On Mon, May 3, 2010 at 11:10 AM, David du Colombier <0intro@gmail.com> wrote:
>>> Does it mean IL has performance issue on long-distance networks?
>>
>> As I understand it, the real problem is that Internet
>> doesn't handle IL well.
>
> They are both problems.  Routing issues aside,
> IL is particularly bad on asymmetric networks like
> cable modems (download much faster than upload).
>

Did anyone experiment with using sliding windows in IL? Could help.

Re: [9fans] Distributed Pipelines

[ron minnich]

On Tue, May 4, 2010 at 3:46 PM, Jorden M <jrm8005@gmail.com> wrote:

> Did anyone experiment with using sliding windows in IL? Could help.

The question I have, based on probably not enough knowledge: how much
of what IL was intended to do is remedied by IPV6? One thing I recall
is that a big problem with v4 was that it did not preserve record
boundaries, which seems won't be an issue in v6. What else did IL
remedy, and how much of what v6 does not remedy is important?

ron

Re: [9fans] Distributed Pipelines

[erik quanstrom]

> The question I have, based on probably not enough knowledge: how much
> of what IL was intended to do is remedied by IPV6? One thing I recall
> is that a big problem with v4 was that it did not preserve record
> boundaries, which seems won't be an issue in v6. What else did IL
> remedy, and how much of what v6 does not remedy is important?

i believe that it is tcp that doesn't preserve record boundaries, not
ip.

- erik

Re: [9fans] Distributed Pipelines

[ron minnich]

On Tue, May 4, 2010 at 11:16 AM, erik quanstrom
<quanstro@labs.coraid.com> wrote:

> i believe that it is tcp that doesn't preserve record boundaries, not
> ip.

Let me rephrase. My understanding is  that tcp on v6 preserves record
boundaries. Is that wrong?

ron

Re: [9fans] Distributed Pipelines

[erik quanstrom]

> On Tue, May 4, 2010 at 11:16 AM, erik quanstrom
> <quanstro@labs.coraid.com> wrote:
>
> > i believe that it is tcp that doesn't preserve record boundaries, not
> > ip.
>
> Let me rephrase. My understanding is  that tcp on v6 preserves record
> boundaries. Is that wrong?

perhaps you mean sctcp?  i don't see any differences in
processing in /sys/src/9/ip/tcp.c that depend on ip4/ip6.

here's wiki's take on the matter:
http://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol
(scroll down to the table)

- erik

Re: [9fans] Distributed Pipelines

[Russ Cox]

TCP doesn't preserve message boundaries.

The pre-9P2000 kernels relied on having a transport
protocol that preserved message boundaries in order
to work one 9P packet at a time with ordinary read calls.
You could work around it by pushing an "fcall" stream module
to reinsert the boundaries on TCP, but it was extra hassle.

In the 9P2000 rewrite, we introduced a read9pmsg function
to the C library and fixed the kernel to do the same, so
that the protocol and its implementations would not
depend on the underlying transport for message boundaries.

So it doesn't matter anymore.  The fix was in the 9P
implementations, not IPv6.

Russ

Re: [9fans] Distributed Pipelines

[erik quanstrom]

> So it doesn't matter anymore.  The fix was in the 9P
> implementations, not IPv6.

it doesn't matter anymore for 9p.  but rx is still broken over tcp.

- erik