[9fans] Issues with 2 networks, fs server, and namespaces

[9fans] Issues with 2 networks, fs server, and namespaces

[Devon H. O'Dell]

Hello all,

I'm trying to set up a group of servers (these are running on VMWare
ESXi, and working great -- CPU server running with two APs, though
adding more causes it to fault with a divide by zero?). Auth server's
got its own 1GB fossil, boots with the 9pcauth kernel. CPU server
boots from a small fossil. Both Auth and CPU are on the public
internet via ether0 so that they are cpu/drawtermable. They do not
boot from the file server because I didn't want to set up a DHCP
server that was connected to the Internet (ISP getting mad and
whatnot). While I've configured the internal network to be on it's own
vswitch (managed through vmware, no real network connectivity), I've
been struggling with the prior configuration enough that I don't want
to just `give up' on it.

The FS, however, sits on a private network. CPU and Auth are connected
to this network via ether1. However, I'm having the following issues:

#1) Using two networks on two different interfaces is a pain in the
ass. I've got:
bind '#l1' /net.alt
bind '#I1' /net.alt

in my /cfg/cpu/namespace. If I simply have them here, ip/ipconfig -N
-x ether1 ether /net.alt/ether1 complains in cpurc about no ip being
attached to /net.alt. So I have to put that in /cfg/cpu/cpurc also. I
don't quite understand why everything's architected to have a single
ip stack on a single ethernet; in this case, it really isn't
convenient that it doesn't determine the correct interface via routing
tables or somesuch. Is there something basic that I'm missing here?

#2) Drawterm is taking forever and a day to connect and log in. It's
either an auth issue or a DNS issue. Best guesses as to what this
could be and how I should go about diagnosing it?

#3) Trying to mount the fileserver globally is elusive. I want to
mount /n/fs/usr over /usr and /n/fs/mail over /mail. Perfectly happy
with that. However:

 o Doing that in cpurc doesn't put it in the global namespace
 o Doing it in /cfg/cpu/namespace doesn't have an ip yet so I can't
run srv /net.alt/tcp!10.0.0.3!9fs in the first place
 o Doing it in /rc/bin/service/tcp17010 causes me to get `cpu:
negotiating authentication method: [public auth server ip]: cs gave
empty translation list'

Mounting it from /n/fs after booting works fine (but it makes me auth,
which is kind of weird -- I guess I need to set up a secstore? -- I
figured that eve would be able to connect without auth, given that
everything's tied to the same auth server, no matter which network
it's on, and that a user drawterming in would be able to connect by
virtue of having authed when connecting in the first place.)

I know the `preferred way' is to boot the CPU server from the
fileserver. While I could feasibly reconfigure my setup to do this,
I'd prefer to figure it out this way first, given the amount of time
I've been banging my head against the wall on it :)

--dho

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[Christopher Nielsen]

You don't need a second IP stack. You can run both interfaces on the
same IP stack and routing will just work. That's how I did it when I
had a similar setup.

-Chris

On Fri, Aug 21, 2009 at 14:07, Devon H. O'Dell<devon.odell@gmail.com> wrote:
> Hello all,
>
> I'm trying to set up a group of servers (these are running on VMWare
> ESXi, and working great -- CPU server running with two APs, though
> adding more causes it to fault with a divide by zero?). Auth server's
> got its own 1GB fossil, boots with the 9pcauth kernel. CPU server
> boots from a small fossil. Both Auth and CPU are on the public
> internet via ether0 so that they are cpu/drawtermable. They do not
> boot from the file server because I didn't want to set up a DHCP
> server that was connected to the Internet (ISP getting mad and
> whatnot). While I've configured the internal network to be on it's own
> vswitch (managed through vmware, no real network connectivity), I've
> been struggling with the prior configuration enough that I don't want
> to just `give up' on it.
>
> The FS, however, sits on a private network. CPU and Auth are connected
> to this network via ether1. However, I'm having the following issues:
>
> #1) Using two networks on two different interfaces is a pain in the
> ass. I've got:
> bind '#l1' /net.alt
> bind '#I1' /net.alt
>
> in my /cfg/cpu/namespace. If I simply have them here, ip/ipconfig -N
> -x ether1 ether /net.alt/ether1 complains in cpurc about no ip being
> attached to /net.alt. So I have to put that in /cfg/cpu/cpurc also. I
> don't quite understand why everything's architected to have a single
> ip stack on a single ethernet; in this case, it really isn't
> convenient that it doesn't determine the correct interface via routing
> tables or somesuch. Is there something basic that I'm missing here?
>
> #2) Drawterm is taking forever and a day to connect and log in. It's
> either an auth issue or a DNS issue. Best guesses as to what this
> could be and how I should go about diagnosing it?
>
> #3) Trying to mount the fileserver globally is elusive. I want to
> mount /n/fs/usr over /usr and /n/fs/mail over /mail. Perfectly happy
> with that. However:
>
>  o Doing that in cpurc doesn't put it in the global namespace
>  o Doing it in /cfg/cpu/namespace doesn't have an ip yet so I can't
> run srv /net.alt/tcp!10.0.0.3!9fs in the first place
>  o Doing it in /rc/bin/service/tcp17010 causes me to get `cpu:
> negotiating authentication method: [public auth server ip]: cs gave
> empty translation list'
>
> Mounting it from /n/fs after booting works fine (but it makes me auth,
> which is kind of weird -- I guess I need to set up a secstore? -- I
> figured that eve would be able to connect without auth, given that
> everything's tied to the same auth server, no matter which network
> it's on, and that a user drawterming in would be able to connect by
> virtue of having authed when connecting in the first place.)
>
> I know the `preferred way' is to boot the CPU server from the
> fileserver. While I could feasibly reconfigure my setup to do this,
> I'd prefer to figure it out this way first, given the amount of time
> I've been banging my head against the wall on it :)
>
> --dho
>
>



-- 
Christopher Nielsen
"They who can give up essential liberty for temporary
safety, deserve neither liberty nor safety." --Benjamin Franklin

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[Devon H. O'Dell]

2009/8/21 Christopher Nielsen <cnielsen@pobox.com>:
> You don't need a second IP stack. You can run both interfaces on the
> same IP stack and routing will just work. That's how I did it when I
> had a similar setup.

I do need a second IP stack because the other network is on another
switch on the other interface, and I do not particularly want to run a
private network over the vswitch hooked to the public internet.

--dho

> -Chris
>
> On Fri, Aug 21, 2009 at 14:07, Devon H. O'Dell<devon.odell@gmail.com> wrote:
>> Hello all,
>>
>> I'm trying to set up a group of servers (these are running on VMWare
>> ESXi, and working great -- CPU server running with two APs, though
>> adding more causes it to fault with a divide by zero?). Auth server's
>> got its own 1GB fossil, boots with the 9pcauth kernel. CPU server
>> boots from a small fossil. Both Auth and CPU are on the public
>> internet via ether0 so that they are cpu/drawtermable. They do not
>> boot from the file server because I didn't want to set up a DHCP
>> server that was connected to the Internet (ISP getting mad and
>> whatnot). While I've configured the internal network to be on it's own
>> vswitch (managed through vmware, no real network connectivity), I've
>> been struggling with the prior configuration enough that I don't want
>> to just `give up' on it.
>>
>> The FS, however, sits on a private network. CPU and Auth are connected
>> to this network via ether1. However, I'm having the following issues:
>>
>> #1) Using two networks on two different interfaces is a pain in the
>> ass. I've got:
>> bind '#l1' /net.alt
>> bind '#I1' /net.alt
>>
>> in my /cfg/cpu/namespace. If I simply have them here, ip/ipconfig -N
>> -x ether1 ether /net.alt/ether1 complains in cpurc about no ip being
>> attached to /net.alt. So I have to put that in /cfg/cpu/cpurc also. I
>> don't quite understand why everything's architected to have a single
>> ip stack on a single ethernet; in this case, it really isn't
>> convenient that it doesn't determine the correct interface via routing
>> tables or somesuch. Is there something basic that I'm missing here?
>>
>> #2) Drawterm is taking forever and a day to connect and log in. It's
>> either an auth issue or a DNS issue. Best guesses as to what this
>> could be and how I should go about diagnosing it?
>>
>> #3) Trying to mount the fileserver globally is elusive. I want to
>> mount /n/fs/usr over /usr and /n/fs/mail over /mail. Perfectly happy
>> with that. However:
>>
>>  o Doing that in cpurc doesn't put it in the global namespace
>>  o Doing it in /cfg/cpu/namespace doesn't have an ip yet so I can't
>> run srv /net.alt/tcp!10.0.0.3!9fs in the first place
>>  o Doing it in /rc/bin/service/tcp17010 causes me to get `cpu:
>> negotiating authentication method: [public auth server ip]: cs gave
>> empty translation list'
>>
>> Mounting it from /n/fs after booting works fine (but it makes me auth,
>> which is kind of weird -- I guess I need to set up a secstore? -- I
>> figured that eve would be able to connect without auth, given that
>> everything's tied to the same auth server, no matter which network
>> it's on, and that a user drawterming in would be able to connect by
>> virtue of having authed when connecting in the first place.)
>>
>> I know the `preferred way' is to boot the CPU server from the
>> fileserver. While I could feasibly reconfigure my setup to do this,
>> I'd prefer to figure it out this way first, given the amount of time
>> I've been banging my head against the wall on it :)
>>
>> --dho
>>
>>
>
>
>
> --
> Christopher Nielsen
> "They who can give up essential liberty for temporary
> safety, deserve neither liberty nor safety." --Benjamin Franklin
>
>

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[Devon H. O'Dell]

2009/8/21 Devon H. O'Dell <devon.odell@gmail.com>:
> 2009/8/21 Christopher Nielsen <cnielsen@pobox.com>:
>> You don't need a second IP stack. You can run both interfaces on the
>> same IP stack and routing will just work. That's how I did it when I
>> had a similar setup.

Wait, I misread your explanation. Would you care to explain more about
that? Is that just binding '#l1' into /net?

--dho

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[Noah Evans]

Hey Devon,

1. Others know more about that than I do. Wait a bit, that problem
might get solved.
2. drawterm tends to hang on secstore for me. Try a bogus -s option or
use a p9p secstore/factotum and see what happens.
3. what's stopping you from setting up your external network as the
one on /net.alt and using dhcp internally? Or booting tcp off the file
server without dhcp? e.g. baking it into plan9.ini, or entering it
manually at boot time? I've tried similar tricks to yours using
inferno and getting cute and breaking the convention of /net.alt for
external networks has always ended in a world of pain.

Noah

On Fri, Aug 21, 2009 at 11:07 PM, Devon H. O'Dell<devon.odell@gmail.com> wrote:
> Hello all,
>
> I'm trying to set up a group of servers (these are running on VMWare
> ESXi, and working great -- CPU server running with two APs, though
> adding more causes it to fault with a divide by zero?). Auth server's
> got its own 1GB fossil, boots with the 9pcauth kernel. CPU server
> boots from a small fossil. Both Auth and CPU are on the public
> internet via ether0 so that they are cpu/drawtermable. They do not
> boot from the file server because I didn't want to set up a DHCP
> server that was connected to the Internet (ISP getting mad and
> whatnot). While I've configured the internal network to be on it's own
> vswitch (managed through vmware, no real network connectivity), I've
> been struggling with the prior configuration enough that I don't want
> to just `give up' on it.
>
> The FS, however, sits on a private network. CPU and Auth are connected
> to this network via ether1. However, I'm having the following issues:
>
> #1) Using two networks on two different interfaces is a pain in the
> ass. I've got:
> bind '#l1' /net.alt
> bind '#I1' /net.alt
>
> in my /cfg/cpu/namespace. If I simply have them here, ip/ipconfig -N
> -x ether1 ether /net.alt/ether1 complains in cpurc about no ip being
> attached to /net.alt. So I have to put that in /cfg/cpu/cpurc also. I
> don't quite understand why everything's architected to have a single
> ip stack on a single ethernet; in this case, it really isn't
> convenient that it doesn't determine the correct interface via routing
> tables or somesuch. Is there something basic that I'm missing here?
>
> #2) Drawterm is taking forever and a day to connect and log in. It's
> either an auth issue or a DNS issue. Best guesses as to what this
> could be and how I should go about diagnosing it?
>
> #3) Trying to mount the fileserver globally is elusive. I want to
> mount /n/fs/usr over /usr and /n/fs/mail over /mail. Perfectly happy
> with that. However:
>
>  o Doing that in cpurc doesn't put it in the global namespace
>  o Doing it in /cfg/cpu/namespace doesn't have an ip yet so I can't
> run srv /net.alt/tcp!10.0.0.3!9fs in the first place
>  o Doing it in /rc/bin/service/tcp17010 causes me to get `cpu:
> negotiating authentication method: [public auth server ip]: cs gave
> empty translation list'
>
> Mounting it from /n/fs after booting works fine (but it makes me auth,
> which is kind of weird -- I guess I need to set up a secstore? -- I
> figured that eve would be able to connect without auth, given that
> everything's tied to the same auth server, no matter which network
> it's on, and that a user drawterming in would be able to connect by
> virtue of having authed when connecting in the first place.)
>
> I know the `preferred way' is to boot the CPU server from the
> fileserver. While I could feasibly reconfigure my setup to do this,
> I'd prefer to figure it out this way first, given the amount of time
> I've been banging my head against the wall on it :)
>
> --dho
>
>

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[Christopher Nielsen]

Certainly. No problem.

Yes, just bind #l1 to /net, assign it an IP addr on that private
network, and put a route in the routing table for good measure. Though
that last step may not be entirely necessary.

-Chris

On Fri, Aug 21, 2009 at 14:59, Devon H. O'Dell<devon.odell@gmail.com> wrote:
> 2009/8/21 Devon H. O'Dell <devon.odell@gmail.com>:
>> 2009/8/21 Christopher Nielsen <cnielsen@pobox.com>:
>>> You don't need a second IP stack. You can run both interfaces on the
>>> same IP stack and routing will just work. That's how I did it when I
>>> had a similar setup.
>
> Wait, I misread your explanation. Would you care to explain more about
> that? Is that just binding '#l1' into /net?
>
> --dho
>
>



--
Christopher Nielsen
"They who can give up essential liberty for temporary
safety, deserve neither liberty nor safety." --Benjamin Franklin

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[Devon H. O'Dell]

2009/8/21 Noah Evans <noah.evans@gmail.com>:
> Hey Devon,
>
> 1. Others know more about that than I do. Wait a bit, that problem
> might get solved.

I think I brought that up because if anybody has ideas about fixing
them or making them better, I would like to do that.

> 2. drawterm tends to hang on secstore for me. Try a bogus -s option or
> use a p9p secstore/factotum and see what happens.

I'm not sure I have secstore set up. Perhaps that's it?

> 3. what's stopping you from setting up your external network as the
> one on /net.alt and using dhcp internally? Or booting tcp off the file
> server without dhcp? e.g. baking it into plan9.ini, or entering it
> manually at boot time? I've tried similar tricks to yours using
> inferno and getting cute and breaking the convention of /net.alt for
> external networks has always ended in a world of pain.

Tried to explain that. I just don't want to due to the number of hours
I've spent trying to configure it this way :). I think Christopher's
suggestion is what I'll do next; I totally forgot that was possible.

--dho

> Noah
>
> On Fri, Aug 21, 2009 at 11:07 PM, Devon H. O'Dell<devon.odell@gmail.com> wrote:
>> Hello all,
>>
>> I'm trying to set up a group of servers (these are running on VMWare
>> ESXi, and working great -- CPU server running with two APs, though
>> adding more causes it to fault with a divide by zero?). Auth server's
>> got its own 1GB fossil, boots with the 9pcauth kernel. CPU server
>> boots from a small fossil. Both Auth and CPU are on the public
>> internet via ether0 so that they are cpu/drawtermable. They do not
>> boot from the file server because I didn't want to set up a DHCP
>> server that was connected to the Internet (ISP getting mad and
>> whatnot). While I've configured the internal network to be on it's own
>> vswitch (managed through vmware, no real network connectivity), I've
>> been struggling with the prior configuration enough that I don't want
>> to just `give up' on it.
>>
>> The FS, however, sits on a private network. CPU and Auth are connected
>> to this network via ether1. However, I'm having the following issues:
>>
>> #1) Using two networks on two different interfaces is a pain in the
>> ass. I've got:
>> bind '#l1' /net.alt
>> bind '#I1' /net.alt
>>
>> in my /cfg/cpu/namespace. If I simply have them here, ip/ipconfig -N
>> -x ether1 ether /net.alt/ether1 complains in cpurc about no ip being
>> attached to /net.alt. So I have to put that in /cfg/cpu/cpurc also. I
>> don't quite understand why everything's architected to have a single
>> ip stack on a single ethernet; in this case, it really isn't
>> convenient that it doesn't determine the correct interface via routing
>> tables or somesuch. Is there something basic that I'm missing here?
>>
>> #2) Drawterm is taking forever and a day to connect and log in. It's
>> either an auth issue or a DNS issue. Best guesses as to what this
>> could be and how I should go about diagnosing it?
>>
>> #3) Trying to mount the fileserver globally is elusive. I want to
>> mount /n/fs/usr over /usr and /n/fs/mail over /mail. Perfectly happy
>> with that. However:
>>
>>  o Doing that in cpurc doesn't put it in the global namespace
>>  o Doing it in /cfg/cpu/namespace doesn't have an ip yet so I can't
>> run srv /net.alt/tcp!10.0.0.3!9fs in the first place
>>  o Doing it in /rc/bin/service/tcp17010 causes me to get `cpu:
>> negotiating authentication method: [public auth server ip]: cs gave
>> empty translation list'
>>
>> Mounting it from /n/fs after booting works fine (but it makes me auth,
>> which is kind of weird -- I guess I need to set up a secstore? -- I
>> figured that eve would be able to connect without auth, given that
>> everything's tied to the same auth server, no matter which network
>> it's on, and that a user drawterming in would be able to connect by
>> virtue of having authed when connecting in the first place.)
>>
>> I know the `preferred way' is to boot the CPU server from the
>> fileserver. While I could feasibly reconfigure my setup to do this,
>> I'd prefer to figure it out this way first, given the amount of time
>> I've been banging my head against the wall on it :)
>>
>> --dho
>>
>>
>
>

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[Devon H. O'Dell]

Well, we're getting somewhere. Using /cfg/cpu/namespace still seems to
do nothing to get ether1 into /net. Putting it into cpurc does the
trick though, go figure.

However, I've got a new issue. When I go to mount the file server, I'm
getting this:

mount: auth_proxy: auth_proxy rpc write: p9sk1@int.9vx.org: no key
matches  proto=p9sk1 dom=int.9vx.org role=client user? !password?
mount: mount /n/fs: fossil authCheck: auth protocol not finished

I mentioned I wasn't sure that I'd set up secstore. I think I really
mean, ``I haven't set up secstore, and I'm not sure how" :)

Tips?

--dho

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[Devon H. O'Dell]

And that's taken care of. Didn't have an authdom configured in
/lib/ndb/local, and for some reason, I forgot to set up keyfs on the
auth server. Thought I had that taken care of.

Of course, I'm now faced with another new issue. auth/debug looks like
it just tries to debug factotum keys. This machine has an interface on
9vx.org and another on int.9vx.org. However, auth/debug only tries to
debug 9vx.org, leading me to believe that factotum has no key for
int.9vx.org. Still getting that auth protocol not finished for
int.9vx.org, but I'm sure if I could get it to put that key in place,
everything would be great.

Ideas?

--dho

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[erik quanstrom]

> mount: auth_proxy: auth_proxy rpc write: p9sk1@int.9vx.org: no key
> matches  proto=p9sk1 dom=int.9vx.org role=client user? !password?
> mount: mount /n/fs: fossil authCheck: auth protocol not finished

that sounds like you are cpu'd into a machine and /mnt/factotum
is the hostowner's factotum, to which you have no permissions.

e.g.

; cpu
ladd; cpu -h minooka.coraid.com
cpu: can't authenticate: minooka.coraid.com: auth_proxy rpc write: p9sk1@coraid.com: no key matches  proto=p9sk1 dom=coraid.com role=client user? !password?


- erik

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[erik quanstrom]

> I'm trying to set up a group of servers (these are running on VMWare
> ESXi, and working great -- CPU server running with two APs, though
> adding more causes it to fault with a divide by zero?). Auth server's

could you be more specific about this?

> The FS, however, sits on a private network. CPU and Auth are connected
> to this network via ether1. However, I'm having the following issues:
>
> #1) Using two networks on two different interfaces is a pain in the
> ass. I've got:
> bind '#l1' /net.alt
> bind '#I1' /net.alt

you want something like this in /lib/namespace.machinewith2ndipstack
 cat /lib/namespace.ladd
bind -b #l1 /net.alt
bind -b #I1 /net.alt
mount -a /srv/cs_net.alt /net.alt
mount -a /srv/dns_net.alt /net.alt
#

and you want something like this in your cpurc

	bind -b '#l1' /net.alt
	bind -b '#I1' /net.alt
	ip/ipconfig -x /net.alt -g 192.168.10.254 ether /net.alt/ether1 add 192.168.10.1  /120
	ndb/cs -x /net.alt -f /lib/ndb/external
	ndb/dns -Rrsx /net.alt -f /lib/ndb/external	#external
- or -
	ndb/dns -s				#internal
	aux/listen -q -t /rc/bin/service.auth -d /rc/bin/service.ext /net.alt/tcp


- erik

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[erik quanstrom]

> Of course, I'm now faced with another new issue. auth/debug looks like
> it just tries to debug factotum keys. This machine has an interface on
> 9vx.org and another on int.9vx.org. However, auth/debug only tries to
> debug 9vx.org, leading me to believe that factotum has no key for
> int.9vx.org. Still getting that auth protocol not finished for
> int.9vx.org, but I'm sure if I could get it to put that key in place,
> everything would be great.

auth domain != dns domain.  there's no reason for them even
to be similar.  (except human convienence.)

- erik

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[erik quanstrom]

On Fri Aug 21 19:55:55 EDT 2009, devon.odell@gmail.com wrote:
> Well, we're getting somewhere. Using /cfg/cpu/namespace still seems to
> do nothing to get ether1 into /net. Putting it into cpurc does the
> trick though, go figure.

you need it in both places, as namespace doesn't apply to the console.

- erik

Re: [9fans] Issues with 2 networks, fs server, and namespaces

[Devon H. O'Dell]

2009/8/21 erik quanstrom <quanstro@coraid.com>:
> On Fri Aug 21 19:55:55 EDT 2009, devon.odell@gmail.com wrote:
>> Well, we're getting somewhere. Using /cfg/cpu/namespace still seems to
>> do nothing to get ether1 into /net. Putting it into cpurc does the
>> trick though, go figure.
>
> you need it in both places, as namespace doesn't apply to the console.

Aha. Ok.

It's working now. Turned out the fs had its authdom in nvram set to
the wrong domain. Oops.

Thanks all!

--dho

> - erik
>
>