[9fans] Multi-stack mail problem.

[9fans] Multi-stack mail problem.

[Dan Cross]

I mentioned this before, perhaps back in January.  My mail server has
two network interfaces, with seperate IP stacks for each.  The internal
stack gets mounted on /net, and the external stack gets mounted on
/net.alt.  However, for the last few months, I've noticed that when I
try to send mail, smtp blocks and hangs trying to send it out the
internal interface (mounted on /net); I've not spent the time to track
down when and where it's hanging, but worked around it by adding:
``bind /net.alt /net'' to /mail/lib/remotemail before invoking smtp.
That seems like an imperfect solution; what if I decide to add a
machine on the internal network that accepts mail via SMTP?  I suppose
I could hack remotemail to try and be smart about binding stacks around
based on where one is sending to, but one would hope for something more
automatic.

Indeed, it seems it used to be different.  In particular, I remember
dhog once saying that if DNS failed, it would try the lookup on
/net.alt, and if that succeeded, use that stack instead.  What's more,
I distinctly remember this working.  On the other hand, Dave said it
was a terrible hack (and indeed, it is).  But Dave had also changed the
IP stack so that if a default route didn't exist on an interface, an
error would immediately be bounced up the stack if a packet couldn't be
sent, and higher-level code (such as that which tries /net.alt) could
take over and try something else.  Only it seems that somewhere along
the way, something broke, and the `try /net.alt next' trick no longer
works.  Lamentable hack though it was, it did serve a purpose, and its
disappearance makes me sad.  :-(  Does anyone know what happened, and
how to either a) `fix' it, or b) do something to give the same effect?
Thanks.

	- Dan C.

Re: [9fans] Multi-stack mail problem.

[David Presotto]

That sounds exactly like out setup.  Dial(2) actually tries first
/net and then /net.alt, which is a terrible hack I did years ago
to deal with out situation.  However it continues to work for us.

I take it that you were relying on exactly the same dial(2) roll
over.  It sounds like the dial used to fail on the inside before
and then get tied on the outside.  Now it sounds like perhaps it
is succeeding on the inside but that where it succeeds to isn't
something that really won't deliver your mail.  Is there perhaps
a system on the inside that is advertising MX entries for everything?

The best thing to do is turn on debugging on smtp (in /mail/lib/remotemail)
and see what's really going on.

Re: [9fans] Multi-stack mail problem.

[Lucio De Re]

On Mon, Sep 29, 2003 at 09:12:04PM -0400, David Presotto wrote:
>
> That sounds exactly like out setup.  Dial(2) actually tries first
> /net and then /net.alt, which is a terrible hack I did years ago
> to deal with out situation.  However it continues to work for us.
>
Would it be out of the question to teach dial to consult the routing
table(s)?  Plan 9 makes networking so painless, it is hard to become
an expert, so questions like this may have obvious answers that are
anything but obvious :-)

++L

Re: [9fans] Multi-stack mail problem.

[Charles Forsyth]

the stacks #I0, #I1, ... are all independent and have their own routing tables.
you don't use separate stacks just to support several physical interfaces.
you use separate stacks to provide complete isolation of each from the others.
thus you still need to control which routing tables dial checked.
furthermore, one or more stacks can be imported from another machine.
furthermore, dial works across a range of protocols including non-internet ones.
routing tables are internet-specific.

thus you still need to search somehow through the name space, not #I...
nor can routing tables as such be used.

on the other hand, {/net /net.alt} only works for dial for two interfaces,
which does suggest that Something Must Be Done.

Re: [9fans] Multi-stack mail problem.

[Lucio De Re]

On Tue, Sep 30, 2003 at 09:20:10AM +0100, Charles Forsyth wrote:
>
> the stacks #I0, #I1, ... are all independent and have their own routing tables.

Good point, bad idea...  I guess IP will require IP routes and other
things user other routes, so maybe it's not such a bad idea.

> you don't use separate stacks just to support several physical interfaces.
> you use separate stacks to provide complete isolation of each from the others.

... I was wondering how one makes a router out of a Plan 9 host.  This
bit I find a little out of the ordinary.

> thus you still need to control which routing tables dial checked.
> furthermore, one or more stacks can be imported from another machine.
> furthermore, dial works across a range of protocols including non-internet ones.
> routing tables are internet-specific.
>
Well, by that name.  Some sort of routing has to exist, no matter the
protocol, at best it's totally trivial.

> thus you still need to search somehow through the name space, not #I...
> nor can routing tables as such be used.
>
So it's a generic solution one needs to find, as you point out.  It
helps if the available information is associated with the interface,
which is a namespace issue.  Perhaps /net/ndb can be used, maybe a
full routing only database?

> on the other hand, {/net /net.alt} only works for dial for two interfaces,
> which does suggest that Something Must Be Done.

It should not be necessary to break the model, from

	net!dest!port

we want to resolve to an interface, which is the leading field, is
it not?  But it is easy to fall into the trap of "all the world is
IP".

++L

Re: [9fans] Multi-stack mail problem.

[boyd, rounin]

> Would it be out of the question to teach dial to consult the routing
> table(s)?

maybe.  i feel you'd be walking into a minefield.

Re: [9fans] Multi-stack mail problem.

[Lucio De Re]

On Tue, Sep 30, 2003 at 10:42:41AM +0200, boyd, rounin wrote:
>
> > Would it be out of the question to teach dial to consult the routing
> > table(s)?
>
> maybe.  i feel you'd be walking into a minefield.

IP without routing is 40 bytes too big in each packet.  May as well
define IL over HDLC (I've been wanting to do this for a long time),
of course I mean Ethernet, and retain 9P over TCP over IP only for
connections via Internet Access Villans.  I'm sure dropping all IP
routing bloat will make the LAN Plan 9 workstation kernel a whole
lot simpler.

We can relegate IP and routing to specialised nodes, as loadable
kernel modules :-)

++L

Re: [9fans] Multi-stack mail problem.

[boyd, rounin]

> But it is easy to fall into the trap of "all the world is IP".

all the world is IP.  many of the problems are _hard_.

Re: [9fans] Multi-stack mail problem.

[Fco.J.Ballesteros]

> connections via Internet Access Villans.  I'm sure dropping all IP
> routing bloat will make the LAN Plan 9 workstation kernel a whole
> lot simpler.

But now that we have the common idiom (i.e. ip) being adopted by
everything in the world, it seems to be a mistake to fall appart
and go back to il. We had lots of problems because our atm vlan
config prior to adopting tcp for 9p connections.

Re: [9fans] Multi-stack mail problem.

[boyd, rounin]

> IP without routing is 40 bytes too big in each packet.

tried ATM?

Re: [9fans] Multi-stack mail problem.

[Lucio De Re]

On Tue, Sep 30, 2003 at 11:16:26AM +0200, Fco.J.Ballesteros wrote:
>
> > connections via Internet Access Villans.  I'm sure dropping all IP
> > routing bloat will make the LAN Plan 9 workstation kernel a whole
> > lot simpler.
>
> But now that we have the common idiom (i.e. ip) being adopted by
> everything in the world, it seems to be a mistake to fall appart
> and go back to il. We had lots of problems because our atm vlan
> config prior to adopting tcp for 9p connections.

Sort of.  There are certainly economies of scale involved, where
a router today costs less than a trivial HDLC controller for the
8-bit PC bus did in 1982, but is it really all for the better?

Now that it's dawning on the trend setters that they've forgotten
to establish the social rules to play by, maybe somebody will also
realise that a large fraction of the "developing world" could make
good use of the obsolete equipment that the developed world is set
on getting rid of, if only one could stop the marketing tide that
insists that only tomorrow's products are any good for the present
circumstances.  And the refrain that suggests that "it costs too
much" to recycle technology when perhaps the correct sentence is
"it would undercut our sales".

Yes, we've definitely slipped up on the ethical end of things.

++L

Re: [9fans] Multi-stack mail problem.

[C H Forsyth]

> But it is easy to fall into the trap of "all the world is IP".

there are indeed many networks, even new ones, that aren't
Internet Protocol, nor should it be necessary (in the Plan9/Inferno environment)
to require an IP stack if you don't need one.

part of the point of dial is to insulate you from many of the assumptions
that some other systems make (including ``it's all IP'')

Re: [9fans] Multi-stack mail problem.

[Lucio De Re]

On Tue, Sep 30, 2003 at 11:16:32AM +0200, boyd, rounin wrote:
>
> > IP without routing is 40 bytes too big in each packet.
>
> tried ATM?

No, I have never gone even near it.  Neither do I know anything
about Ethernet, to be perfectly honest.  I had a brush with HDLC,
a long time ago, and it looked quite adequate to my uneducated eye.

I guess it would not serve wireless very well, but seeing it
reintroduced in PPP had me curious.

++L

Re: [9fans] Multi-stack mail problem.

[C H Forsyth]

>>and go back to il. We had lots of problems because our atm vlan
>>config prior to adopting tcp for 9p connections.

that one can't reliably add new IP protocols and have them work
on the Internet in practice is fairly pathetic.
worse, as far as i can tell, IPv6 replicated many of the mistakes of v4!

Re: [9fans] Multi-stack mail problem.

[Geoff Collyer]

those weren't mistakes, they were time-tested and proven practices!
like 7-bit smtp....

Re: [9fans] Multi-stack mail problem.

[Geoff Collyer]

oh, and NAPT is what seemed to mess up IL for me.  IP itself doesn't
contain port numbers and, as i recall, IL put its port numbers at
different offsets in the packet than TCP and UDP did.

Re: [9fans] Multi-stack mail problem.

[Geoff Collyer]

of course, IL ran on top of IP, but never mind.

i believe that one can still write `add net1 net2' messages to
/net*/cs to tell the connection servers which of the available
interfaces to try, and in what order.  perhaps instead of separate
connection servers for each stack (/net, /net.alt, etc.), we want one
connection server that scans /net*/*/clone instead of /net/*/clone at
startup, and can (also) be told

	add /net/tcp /net.alt/tcp

or perhaps an inside connection server and an `inside and outside'
server that one could bind on top of /net in some windows.

incidentally, one problem with `net!dest!service' is that the service
and destination names depend upon the network, so although
`dk!nj/astro/helix!smtp' and `tcp!helix.cs.bell-labs.com!smtp' might
be valid names, one can't easily try the `other' network by swapping
`dk' and `tcp', since `tcp!nj/astro/helix' isn't a meaningful name,
nor is (was) `dk!helix.cs.bell-labs.com', so to use the search
facility of cs across dissimilar networks, it's necessary to define
short names carefully in ndb and use only those.

Re: [9fans] Multi-stack mail problem.

[Bruce Ellis]

froggie uses BPMP (brucee's pci mailbox protocol) to serve the
subservient frog legs.  it screams.  dial is the way to go.  i have
services that bind a dir onto /net and get dialed just fine.

brucee
----- Original Message -----
From: "C H Forsyth" <forsyth@vitanuova.com>
To: <9fans@cse.psu.edu>
Sent: Tuesday, September 30, 2003 7:38 PM
Subject: Re: [9fans] Multi-stack mail problem.


> >>and go back to il. We had lots of problems because our atm vlan
> >>config prior to adopting tcp for 9p connections.
>
> that one can't reliably add new IP protocols and have them work
> on the Internet in practice is fairly pathetic.
> worse, as far as i can tell, IPv6 replicated many of the mistakes of v4!

Re: [9fans] Multi-stack mail problem.

[boyd, rounin]

> > tried ATM?
>
> No, I have never gone even near it.

quick summary:  53 byte packets, 5 byte header, 48 byte payload.

ATM is like X.25 on crack.

do the math.

Re: [9fans] Multi-stack mail problem.

[Geoff Collyer]

One view is that ATM is the less-interesting parts of Datakit (small
cells, but no in-network authentication nor name service).  In a dial
string like `dk!nj/astro/helix!smtp', one handed `nj/astro/helix' to
one's Datakit interface (I forget how the service was passed); there
was none of this ARP and insecure DNS rubbish.

PPP is a classic of second-system effect (see Brooks).  SLIP was
simple but didn't provide for transporting DECnet frames nor BISYNC
nor OSI nor lots of other now-dead or at least not-very-well protocols
(`i feel happy, i feel happy').  So rather than just adding support
for ICMP or whatever was missing but actually desirable, and just
clubbing the other protocols over the head and throwing their
carcasses on the wagon, the standards people got involved,
and decided that PPP needed oodles of negotiation and Assinine One (ASN.1)
obfuscation and other needless complexity.
(The negotiation isn't even guaranteed to terminate, but details, details.)

Re: [9fans] Multi-stack mail problem.

[C H Forsyth]

[-- Attachment #1: Type: text/plain, Size: 13 bytes --]

absolutely.

[-- Attachment #2: Type: message/rfc822, Size: 2991 bytes --]

From: "Bruce Ellis" <brucee@chunder.com>
To: <9fans@cse.psu.edu>
Subject: Re: [9fans] Multi-stack mail problem.
Date: Tue, 30 Sep 2003 19:48:16 +1000
Message-ID: <009701c38738$06a39cd0$c901a8c0@cc77109e>

froggie uses BPMP (brucee's pci mailbox protocol) to serve the
subservient frog legs.  it screams.  dial is the way to go.  i have
services that bind a dir onto /net and get dialed just fine.

brucee
----- Original Message -----
From: "C H Forsyth" <forsyth@vitanuova.com>
To: <9fans@cse.psu.edu>
Sent: Tuesday, September 30, 2003 7:38 PM
Subject: Re: [9fans] Multi-stack mail problem.


> >>and go back to il. We had lots of problems because our atm vlan
> >>config prior to adopting tcp for 9p connections.
>
> that one can't reliably add new IP protocols and have them work
> on the Internet in practice is fairly pathetic.
> worse, as far as i can tell, IPv6 replicated many of the mistakes of v4!

Re: [9fans] Multi-stack mail problem.

[boyd, rounin]

> (The negotiation isn't even guaranteed to terminate, but details, details.)

yes, hacked the 1st Ed PP server and THEN i read the RFC.
i has no guarantee of termination.  what !?!

SLIP seems like a BSD-ism that was hacked up in a night; Simple
Lousy Incompetent Protocol.

dial is a good thing.  it removes all that horrible 'how the hell
do i set up this call' problem.

datakit: not known to work reliably at any speed, but the code is huge ;)

Re: [9fans] Multi-stack mail problem.

[boyd, rounin]

ATM 'crankback' is interesting, but you need it 'cos of the 5 byte
header, which is kinda self defeating.

Re: [9fans] Multi-stack mail problem.

[C H Forsyth]

>>and decided that PPP needed oodles of negotiation and Assinine One (ASN.1)

there's asn.1 in ppp?  i've forgotten (or i never had to do that bit)

Re: [9fans] Multi-stack mail problem.

[Lucio De Re]

On Tue, Sep 30, 2003 at 07:48:16PM +1000, Bruce Ellis wrote:
>
> froggie uses BPMP (brucee's pci mailbox protocol) to serve the
> subservient frog legs.  it screams.  dial is the way to go.  i have
> services that bind a dir onto /net and get dialed just fine.
>
I detect a degree of consensus: dial is where the routing needs to
be determined.  Now we need to figure out how to tech dial all
sorts of independent tricks to establish the relationships between
destinations and interfaces (network stacks).

I don't think we need to worry unduly about the reverse: once a
packet has arrived over an interface, it can be assumed that that
interface is the right one for the connection.  If it can't be,
one can make special arrangements.

*** disclaimer - loose cannon below :-) ***

Maybe we can have a "reach" file or control command in each protocol
stack that returns a positive or negative response when queried as
to whether a given "address" can be reached through that interface.
How clever such a facility could be is for better designers than
I to figure out.

I imagine that some "hints" could be added to speed things up whenever
possible.

++L

Re: [9fans] Multi-stack mail problem.

[C H Forsyth]

[-- Attachment #1: Type: text/plain, Size: 158 bytes --]

i'm not sure it would have helped (much).  the checksums are in different
locations even between tcp and udp.  why must you change the checksum?
well ....

[-- Attachment #2: Type: message/rfc822, Size: 2032 bytes --]

From: Geoff Collyer <geoff@collyer.net>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] Multi-stack mail problem.
Date: Tue, 30 Sep 2003 02:41:33 -0700
Message-ID: <4447df5b1d0853c0dd0f099400dd8b95@collyer.net>

oh, and NAPT is what seemed to mess up IL for me.  IP itself doesn't
contain port numbers and, as i recall, IL put its port numbers at
different offsets in the packet than TCP and UDP did.

Re: [9fans] Multi-stack mail problem.

[boyd, rounin]

> there's asn.1 in ppp?  i've forgotten (or i never had to do that bit)

not iirc.  i think it's just a gramatical problem.

Re: [9fans] Multi-stack mail problem.

[boyd, rounin]

> I detect a degree of consensus: dial is where the routing needs to
> be determined.  Now we need to figure out how to tech dial all
> sorts of independent tricks to establish the relationships between
> destinations and interfaces (network stacks).

hard

> I don't think we need to worry unduly about the reverse: once a
> packet has arrived over an interface, it can be assumed that that
> interface is the right one for the connection.  If it can't be,
> one can make special arrangements.

the interface my die.  the net/route may go down.  there my be mutiple
interfaces on the same net.  no,  this is the 'ultrix school of coding
to fix improbable kernel conditions', which can't be tested.

it goes:

    error 1

        try to fix error 1

        error 2

           try to fix error 2

           error 3

                 ...

Re: [9fans] Multi-stack mail problem.

[boyd, rounin]

checksums?  we don't need no stinkin' [UDP] checksums -- Sun

Re: [9fans] Multi-stack mail problem.

[ron minnich]

On Tue, 30 Sep 2003, boyd, rounin wrote:

> checksums?  we don't need no stinkin' [UDP] checksums -- Sun

well and lots of other people who have been fooled. HP, for example, had a
'smart NIC' in the 80s. Worked great. Except you couldn't ftp certain
files.

And then there was atm. and hippi 800. and myrinet. and quadrics. I'm sure
infiniband will make the same mistake. People keep forgetting: end to end
does NOT mean card-to-card.

We just saw this the other day on myrinet: a certain data pattern would
induce a problem in one out of 2048 links, on one point-to-point link, on
one tcp port. Fun to diagnose, as the symptom was a stuck tcp queue. It
was a bad switch port.

ah well. Networks (and network designers) should stop trying to make
promises they can't keep.

ron

Re: [9fans] Multi-stack mail problem.

[C H Forsyth]

[-- Attachment #1: Type: text/plain, Size: 159 bytes --]

sorry, that wasn't what i meant.  having the
checksums on the data is all very well,
for the reasons mentioned.
why do you need to change them?
well ....

[-- Attachment #2: Type: message/rfc822, Size: 2208 bytes --]

From: "boyd, rounin" <boyd@insultant.net>
To: <9fans@cse.psu.edu>
Subject: Re: [9fans] Multi-stack mail problem.
Date: Tue, 30 Sep 2003 19:09:32 +0200
Message-ID: <021501c38775$a06801c0$b9844051@insultant.net>

checksums?  we don't need no stinkin' [UDP] checksums -- Sun

Re: [9fans] Multi-stack mail problem.

[Geoff Collyer]

Odd, I remember a PPP implementor telling me years ago that he'd had
to implement ASN.1 as part of a from-scratch PPP implementation, yet I
see no reference to ASN.1 in the Plan 9 PP*P* code and only this
reference in the PPP RFCs (in the PPP CHAP RFC):

      The Name field is one or more octets representing the
      identification of the system transmitting the packet.  There are
      no limitations on the content of this field.  For example, it MAY
      contain ASCII character strings or globally unique identifiers in
      ASN.1 syntax.  The Name should not be NUL or CR/LF terminated.
      The size is determined from the Length field.

Maybe the people he was working for decided to use ASN.1 here for some
perverse reason.

Nevertheless, PPP is the sendmail of serial line framing protocols.

Re: [9fans] Multi-stack mail problem.

[Charles Forsyth]

>>Odd, I remember a PPP implementor telling me years ago that he'd had
>>to implement ASN.1 as part of a from-scratch PPP implementation, yet I

i'm sure someone has or will find a use for it somewhere.
if they'd done it today it would use XML for negotiation.

Re: [9fans] Multi-stack mail problem.

[Fco.J.Ballesteros]

[-- Attachment #1: Type: text/plain, Size: 257 bytes --]

Actually, I don't think dial could be teached at all.
It depends on user preferences. Perhaps a sort of
microlenguage to program such preferences could be
the way to go (together with reasonable defaults so you
could forget about it if you dont mind).

[-- Attachment #2: Type: message/rfc822, Size: 2891 bytes --]

From: "boyd, rounin" <boyd@insultant.net>
To: <9fans@cse.psu.edu>
Subject: Re: [9fans] Multi-stack mail problem.
Date: Tue, 30 Sep 2003 19:07:31 +0200
Message-ID: <020101c38775$5a0c8840$b9844051@insultant.net>

> I detect a degree of consensus: dial is where the routing needs to
> be determined.  Now we need to figure out how to tech dial all
> sorts of independent tricks to establish the relationships between
> destinations and interfaces (network stacks).

hard

> I don't think we need to worry unduly about the reverse: once a
> packet has arrived over an interface, it can be assumed that that
> interface is the right one for the connection.  If it can't be,
> one can make special arrangements.

the interface my die.  the net/route may go down.  there my be mutiple
interfaces on the same net.  no,  this is the 'ultrix school of coding
to fix improbable kernel conditions', which can't be tested.

it goes:

    error 1

        try to fix error 1

        error 2

           try to fix error 2

           error 3

                 ...

Re: [9fans] Multi-stack mail problem.

[David Presotto]

 From the comments, I see that some people are confused.  My
terrible hack is only to live with totally separate stacks.
If you want to openly route between different interfaces, or
just have different interfaces to the 'open' internet, there's
no reason for separate stacks.  Any stack can have as many
interfaces as you'ld like.  The record so far was on the
path star switch which support 256 interfaces on a single
stack.

In addition to multiple stacks, there are also multiple mount
points.  More than one network type can be on a single mount
point.  For example, you can have and IP stack and a datakit
stack unioned on the same mount point.  'cs' is artificially
intelligent and tries to figure out which you mean by the name
you give it.  If the name is possible on multiple network
types, it gives you back the union of possibilities for you
to work through.  We lived that way for a long time while
datakit was reliable and IP wasn't.  Eventually datakit
just died to be replaced by the monstrosity called ATM
to later be replaced by the super-monstrosity called
MPLS...  Nothing against wasting bandwidth (it makes money
for our company) just against the amazingly complex connection
setup in those two networks.  They are to datakit, what PPP
is to SLIP.

Clearly, dial doing what it does is a mistake, though I
doubt it is Dan's problem.  I could have made
cs smart enough that I don't nead one per network mount
point but instead one per system that understands all the
mount points.  That way it could consult all the stacks/mount
points and see what is possible.  The reason for not doing that
to date is that the cs doesn't know which mount points/stacks
the user process has in its name space.  If I import an outside
stack from our outside machine, there's no way for my cs
to know that or even have access to that stack.  Having cs under the
mount point makes that a non-problem but pushes the responsibility
to the library.

To wit, I don't have a good idea for the correct solution and
continue the hack until someone comes up with one.