[9fans] $smtp dns failure

[9fans] $smtp dns failure

[Matthias Teege]

Moin,

is there any change in the behavior of /mail/lib/rewrite? I use

([^!]*)!(.*) "/mail/lib/qmail '\s' 'net!$smtp'" "'\2@\1'"

to send email over the local gateway but got

crn Jan 15 16:38:27 dns: dns failure (net!$smtp)

There is a smtp gateway specified in /lib/ndb/local. Do I need to put
the gateway in rewrite?

Matthias

Re: [9fans] $smtp dns failure

[Federico Benavento]

hola,

I have "smtp=smpt.gmail.com" on my profile.

On 1/15/07, Matthias Teege <mteege@gmail.com> wrote:
> Moin,
>
> is there any change in the behavior of /mail/lib/rewrite? I use
>
> ([^!]*)!(.*) "/mail/lib/qmail '\s' 'net!$smtp'" "'\2@\1'"
>
> to send email over the local gateway but got
>
> crn Jan 15 16:38:27 dns: dns failure (net!$smtp)
>
> There is a smtp gateway specified in /lib/ndb/local. Do I need to put
> the gateway in rewrite?
>
> Matthias
>


--
Federico G. Benavento

Re: [9fans] $smtp dns failure

[geoff]

What does

	ndb/csquery net!$smtp!smtp

print?

Re: [9fans] $smtp dns failure

[Russ Cox]

> I have "smtp=smpt.gmail.com" on my profile.

Unfortunately, in the context of dial strings,
there is a difference between the environment
variable smtp and the network variable smtp.

Setting the environment variable in your profile
should not affect /mail/lib/rewrite at all.  It is the
network variable smtp that is named by
the rewrite rule.  For that to work you need to
have set smtp= in the /lib/ndb entry for your
host or in the entry for a network containing it.

> What does
>
>       ndb/csquery net!$smtp!smtp
>
> print?

For the same reason, you actually want to run
    ndb/csquery 'net!$smtp!smtp'
(with quotes), so that the shell does not substitute
the environment variable.

Russ

Re: [9fans] $smtp dns failure

[Matthias Teege]

On 1/15/07, geoff@plan9.bell-labs.com <geoff@plan9.bell-labs.com> wrote:
>         ndb/csquery net!$smtp!smtp

> net!$smtp!smtp
/net/tcp/clone 192.168.0.22!25

which looks good for me. I also tried to put the ip in "rewrite" but
it doesn't help:

crn Jan 15 19:48:59 dns: dns failure (net!$smtp)
crn Jan 15 19:48:59 dns: dns failure (net!192.168.0.22)
crn Jan 15 19:48:59 dns: dns failure (tcp!192.168.0.22)
crn Jan 15 19:48:59 dns: dns failure (192.168.0.22)
crn Jan 15 19:48:59 dns: dns failure (net!192.168.0.22!smtp)

here the csquery output

> tcp!192.168.0.22!smtp
/net/tcp/clone 192.168.0.22!25
> net!192.168.0.22!smtp
/net/tcp/clone 192.168.0.22!25
> 192.168.0.22!smtp
/net/192.168.0.22/clone smtp

Matthias

Re: [9fans] $smtp dns failure

[Steve Simon]

re smtp dns problems:

Hi,

Debugging smtp: check the exact way upas/smtp is being envoked.
Look at the end of /sys/mail/runq, here is an example from mine:

	felix Jan 12 19:33:36 steve/C.095795: execing '/mail/lib/remotemail' 'smtp' 'steve' 'net!$smtp' '9fans@cse.psu.edu'

Then you can try running the remotemail script with rc -x to see
what it is doing, in my example:

	rc -x '/mail/lib/remotemail' 'smtp' 'steve' 'net!$smtp' '9fans@cse.psu.edu'
    ...
	exec /bin/upas/smtp -g 'net!$smtp' -h quintile.net 'net!$smtp' steve 9fans@cse.psu.edu

Finally you can exec smtp with the -d option (debug) to see the SMTP conversation,
and perhaps more importantly the DNS lookups (and failures) that occur.

----------------------

I used to put this in my /mail/lib/remotemail:

	exec /bin/upas/smtp -h $fd $addr $sender $*

But due to a weird bug in the DNS cache (that I couldn't find even after
several attempts) mail would fail, so I changed (at Russ's advice) to

	exec /bin/upas/smtp -g $addr -h $fd $addr $sender $*

and email has worked fine since.

-Steve

Re: [9fans] $smtp dns failure

[Matthias Teege]

On 1/16/07, Steve Simon <steve@quintile.net> wrote:

> Finally you can exec smtp with the -d option (debug) to see the SMTP conversation,
> and perhaps more importantly the DNS lookups (and failures) that occur.

Yes, and it looks like smtp look for an mx record of the smtpserver

exec /bin/upas/smtp -d -h crn.mteege.de 'net!$smtp' mtg mteege@gmail.com
expanding /net!$smtp
sending /net/dns '192.168.0.22 mx'
dns: dns: dns failure

There is no mx record for 192.168.0.22. If I use 'mteege.de', which
has a mx record (192.168.0.22), sending mail works.

I'm not sure if the behavior of upas/smtp is correct.

Matthias

Re: [9fans] $smtp dns failure

[Steve Simon]

> exec /bin/upas/smtp -d -h crn.mteege.de 'net!$smtp' mtg mteege@gmail.com
> expanding /net!$smtp
> sending /net/dns '192.168.0.22 mx'
> dns: dns: dns failure

> There is no mx record for 192.168.0.22. If I use 'mteege.de', which
> has a mx record (192.168.0.22), sending mail works.

Strange, where does it get this IP address from?
do you have smtp=192.168.0.22 in /lib/ndb/local?

The smtp= should point to a FQDN rather than an IP address
to allow upas/smtp to look up the MX and then A RRs.

I get this:

	exec /bin/upas/smtp -d -g 'net!$smtp' -h quintile.net 'net!$smtp' steve steve.simon@mywork.com
	expanding /net!$smtp
	sending /net/dns 'smtp.myisp.com mx'
	dns: dns: resource does not exist
	mxdial trying /net/net!smtp.myisp.com!smtp
	220 ESMTP server ready

-Steve

Re: [9fans] $smtp dns failure

[erik quanstrom]

there is an interesting case  with mxlookup1() in mxdial.c.
if a dns /failure/ is reported for the mx lookup, then mxdial doesn't
try a/cname records.  i've had trouble in the past with mx failures when the
a record looks up just fine.  i'm not sure if the standard says one must quit
on dns failure.

- erik

On Tue Jan 16 13:25:49 EST 2007, steve@quintile.net wrote:
> > exec /bin/upas/smtp -d -h crn.mteege.de 'net!$smtp' mtg mteege@gmail.com
> > expanding /net!$smtp
> > sending /net/dns '192.168.0.22 mx'
> > dns: dns: dns failure
>
> > There is no mx record for 192.168.0.22. If I use 'mteege.de', which
> > has a mx record (192.168.0.22), sending mail works.
>
> Strange, where does it get this IP address from?
> do you have smtp=192.168.0.22 in /lib/ndb/local?
>
> The smtp= should point to a FQDN rather than an IP address
> to allow upas/smtp to look up the MX and then A RRs.
>
> I get this:
>
> 	exec /bin/upas/smtp -d -g 'net!$smtp' -h quintile.net 'net!$smtp' steve steve.simon@mywork.com
> 	expanding /net!$smtp
> 	sending /net/dns 'smtp.myisp.com mx'
> 	dns: dns: resource does not exist
> 	mxdial trying /net/net!smtp.myisp.com!smtp
> 	220 ESMTP server ready
>
> -Steve

Re: [9fans] $smtp dns failure

[Russ Cox]

> there is an interesting case  with mxlookup1() in mxdial.c.
> if a dns /failure/ is reported for the mx lookup, then mxdial doesn't
> try a/cname records.  i've had trouble in the past with mx failures when the
> a record looks up just fine.  i'm not sure if the standard says one must quit
> on dns failure.

if there is a dns failure, the logic there is that since the
mx could not be determined due to some transient dns error,
we are not going to blindly try the underlying host name,
because that is likely to be wrong.  i am sure that i put this
in because of actual problems.

the real problem is that smtp is doing a dns lookup on 192.168.0.22,
which ndb/dns interprets as a request for a reverse lookup
(that is, it looks up 22.0.168.192.in-addr.arpa), which fails.
mxlookup should probably check whether ds->host is a
valid ip address all by itself, and then return 0 if so, rather
than ask dns anything at all.  unfortunately i don't think there
is a library routine that will tell us whether a string is a valid
ip address.  parseip doesn't.

russ

Re: [9fans] $smtp dns failure

[lucio]

[-- Attachment #1: Type: text/plain, Size: 391 bytes --]

> unfortunately i don't think there
> is a library routine that will tell us whether a string is a valid
> ip address.

I had to enhance the APE "bsd" library to implement the OpenLDAP
client tools.  I'm sure importing inet_pton.c from NetBSD to Plan 9
native will not be much more effort.  I do believe it loses the
ability to understand 127.1, but I have not verified this.

++L

[-- Attachment #2: inet_pton.c --]
[-- Type: text/plain, Size: 11071 bytes --]

/*      $NetBSD: inet_pton.c,v 1.16 2000/02/07 18:51:02 itojun Exp $    */

/* Copyright (c) 1996 by Internet Software Consortium.
 *
 * Permission to use, copy, modify, and distribute this software for any
 * purpose with or without fee is hereby granted, provided that the above
 * copyright notice and this permission notice appear in all copies.
 *
 * THE SOFTWARE IS PROVIDED "AS IS" AND INTERNET SOFTWARE CONSORTIUM DISCLAIMS
 * ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES
 * OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL INTERNET SOFTWARE
 * CONSORTIUM BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
 * DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
 * PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
 * ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS
 * SOFTWARE.
 */

#include <sys/param.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>

#include <assert.h>
#include <ctype.h>
#include <errno.h>
#include <stdlib.h>
#include <string.h>

/*
 * WARNING: Don't even consider trying to compile this on a system where
 * sizeof(int) < 4.  sizeof(int) > 4 is fine; all the world's not a VAX.
 */

static int      inet_pton4 (const char *src, unsigned char *dst, int pton);
static int      inet_pton6 (const char *src, unsigned char *dst);

/* int
 * inet_pton(af, src, dst)
 *      convert from presentation format (which usually means ASCII printable)
 *      to network format (which is usually some kind of binary format).
 * return:
 *      1 if the address was valid for the specified address family
 *      0 if the address wasn't valid (`dst' is untouched in this case)
 *      -1 if some other error occurred (`dst' is untouched in this case, too)
 * author:
 *      Paul Vixie, 1996.
 */
int
inet_pton(af, src, dst)
        int af;
        const char *src;
        void *dst;
{
        switch (af) {
        case AF_INET:
                return (inet_pton4(src, dst, 1));
        case AF_INET6:
                return (inet_pton6(src, dst));
        default:
                errno = EAFNOSUPPORT;
                return (-1);
        }
        /* NOTREACHED */
}

/* int
 * inet_pton4(src, dst, pton)
 *      when last arg is 0: inet_aton(). with hexadecimal, octal and shorthand.
 *      when last arg is 1: inet_pton(). decimal dotted-quad only.
 * return:
 *      1 if `src' is a valid input, else 0.
 * notice:
 *      does not touch `dst' unless it's returning 1.
 * author:
 *      Paul Vixie, 1996.
 */
#define INADDRSZ (4)
static int
inet_pton4(src, dst, pton)
        const char *src;
        unsigned char *dst;
        int pton;
{
        unsigned int val;
        unsigned int digit;
        int base, n;
        unsigned char c;
        unsigned int parts[4];
        unsigned int *pp = parts;

        c = *src;
        for (;;) {
                /*
                 * Collect number up to ``.''.
                 * Values are specified as for C:
                 * 0x=hex, 0=octal, isdigit=decimal.
                 */
                if (!isdigit(c))
                        return (0);
                val = 0; base = 10;
                if (c == '0') {
                        c = *++src;
                        if (c == 'x' || c == 'X')
                                base = 16, c = *++src;
                        else if (isdigit(c) && c != '9')
                                base = 8;
                }
                /* inet_pton() takes decimal only */
                if (pton && base != 10)
                        return (0);
                for (;;) {
                        if (isdigit(c)) {
                                digit = c - '0';
                                if (digit >= base)
                                        break;
                                val = (val * base) + digit;
                                c = *++src;
                        } else if (base == 16 && isxdigit(c)) {
                                digit = c + 10 - (islower(c) ? 'a' : 'A');
                                if (digit >= 16)
                                        break;
                                val = (val << 4) | digit;
                                c = *++src;
                        } else
                                break;
                }
                if (c == '.') {
                        /*
                         * Internet format:
                         *      a.b.c.d
                         *      a.b.c   (with c treated as 16 bits)
                         *      a.b     (with b treated as 24 bits)
                         *      a       (with a treated as 32 bits)
                         */
                        if (pp >= parts + 3)
                                return (0);
                        *pp++ = val;
                        c = *++src;
                } else
                        break;
        }
        /*
         * Check for trailing characters.
         */
        if (c != '\0' && !isspace(c))
                return (0);
        /*
         * Concoct the address according to
         * the number of parts specified.
         */
        n = pp - parts + 1;
        /* inet_pton() takes dotted-quad only.  it does not take shorthand. */
        if (pton && n != 4)
                return (0);
        switch (n) {

        case 0:
                return (0);             /* initial nondigit */

        case 1:                         /* a -- 32 bits */
                break;

        case 2:                         /* a.b -- 8.24 bits */
                if (parts[0] > 0xff || val > 0xffffff)
                        return (0);
                val |= parts[0] << 24;
                break;

        case 3:                         /* a.b.c -- 8.8.16 bits */
                if ((parts[0] | parts[1]) > 0xff || val > 0xffff)
                        return (0);
                val |= (parts[0] << 24) | (parts[1] << 16);
                break;

        case 4:                         /* a.b.c.d -- 8.8.8.8 bits */
                if ((parts[0] | parts[1] | parts[2] | val) > 0xff)
                        return (0);
                val |= (parts[0] << 24) | (parts[1] << 16) | (parts[2] << 8);
                break;
        }
        if (dst) {
                val = htonl(val);
                memcpy(dst, &val, INADDRSZ);
        }
        return (1);
}

/* int
 * inet_pton6(src, dst)
 *      convert presentation level address to network order binary form.
 * return:
 *      1 if `src' is a valid [RFC1884 2.2] address, else 0.
 * notice:
 *      (1) does not touch `dst' unless it's returning 1.
 *      (2) :: in a full address is silently ignored.
 * credit:
 *      inspired by Mark Andrews.
 * author:
 *      Paul Vixie, 1996.
 */
#define	IN6ADDRSZ	(16)
static int
inet_pton6(src, dst)
        const char *src;
        unsigned char *dst;
{
        static const char xdigits_l[] = "0123456789abcdef",
                          xdigits_u[] = "0123456789ABCDEF";
        unsigned char tmp[IN6ADDRSZ], *tp, *endp, *colonp;
        const char *xdigits, *curtok;
        int ch, saw_xdigit;
        unsigned int val;

        memset((tp = tmp), '\0', IN6ADDRSZ);
        endp = tp + IN6ADDRSZ;
        colonp = NULL;
        /* Leading :: requires some special handling. */
        if (*src == ':')
                if (*++src != ':')
                        return (0);
        curtok = src;
        saw_xdigit = 0;
        val = 0;
        while ((ch = *src++) != '\0') {
                const char *pch;

                if ((pch = strchr((xdigits = xdigits_l), ch)) == NULL)
                        pch = strchr((xdigits = xdigits_u), ch);
                if (pch != NULL) {
                        val <<= 4;
                        val |= (pch - xdigits);
                        if (val > 0xffff)
                                return (0);
                        saw_xdigit = 1;
                        continue;
                }
                if (ch == ':') {
                        curtok = src;
                        if (!saw_xdigit) {
                                if (colonp)
                                        return (0);
                                colonp = tp;
                                continue;
                        } else if (*src == '\0')
                                return (0);
                        if (tp + 2 > endp)
                                return (0);
                        *tp++ = (unsigned char) (val >> 8) & 0xff;
                        *tp++ = (unsigned char) val & 0xff;
                        saw_xdigit = 0;
                        val = 0;
                        continue;
                }
                if (ch == '.' && ((tp + INADDRSZ) <= endp) &&
                    inet_pton4(curtok, tp, 1) > 0) {
                        tp += INADDRSZ;
                        saw_xdigit = 0;
                        break;  /* '\0' was seen by inet_pton4(). */
                }
                return (0);
        }
        if (saw_xdigit) {
                if (tp + 2 > endp)
                        return (0);
                *tp++ = (unsigned char) (val >> 8) & 0xff;
                *tp++ = (unsigned char) val & 0xff;
        }
        if (colonp != NULL) {
                /*
                 * Since some memmove()'s erroneously fail to handle
                 * overlapping regions, we'll do the shift by hand.
                 */
                const int n = tp - colonp;
                int i;

                if (tp == endp)
                        return (0);
                for (i = 1; i <= n; i++) {
                        endp[- i] = colonp[n - i];
                        colonp[n - i] = 0;
                }
                tp = endp;
        }
        if (tp != endp)
                return (0);
        memcpy(dst, tmp, IN6ADDRSZ);
        return (1);
}

/*
 * Ascii internet address interpretation routine.
 * The value returned is in network order.
 */
unsigned long
inet_addr(cp)
        register const char *cp;
{
        struct in_addr val;

        if (inet_pton4(cp, (unsigned char *)(void *)&val.s_addr, 0))
                return (val.s_addr);
        return (INADDR_NONE);
}

/*
 * Check whether "cp" is a valid ascii representation
 * of an Internet address and convert to a binary address.
 * Returns 1 if the address is valid, 0 if not.
 * This replaces inet_addr, the return value from which
 * cannot distinguish between failure and a local broadcast address.
 */
int
inet_aton(cp, addr)
        register const char *cp;
        struct in_addr *addr;
{
        return inet_pton4(cp, (unsigned char *)(void *)&addr->s_addr, 0);
}

Re: [9fans] $smtp dns failure

[erik quanstrom]

§5 of rfc 2821 spells out how to do the lookup.  smtpd should do that.
if there are any huristics to apply while running the lookup, it
would make sense to me to do as sendmail/postifix/qmail do.
smtp is an interface to the world.  being different isn't a virtue
in this case.

- erik

On Tue Jan 16 13:46:09 EST 2007, rsc@swtch.com wrote:
> > there is an interesting case  with mxlookup1() in mxdial.c.
> > if a dns /failure/ is reported for the mx lookup, then mxdial doesn't
> > try a/cname records.  i've had trouble in the past with mx failures when the
> > a record looks up just fine.  i'm not sure if the standard says one must quit
> > on dns failure.
> 
> if there is a dns failure, the logic there is that since the
> mx could not be determined due to some transient dns error,
> we are not going to blindly try the underlying host name,
> because that is likely to be wrong.  i am sure that i put this
> in because of actual problems.
> 
> the real problem is that smtp is doing a dns lookup on 192.168.0.22,
> which ndb/dns interprets as a request for a reverse lookup
> (that is, it looks up 22.0.168.192.in-addr.arpa), which fails.
> mxlookup should probably check whether ds->host is a
> valid ip address all by itself, and then return 0 if so, rather
> than ask dns anything at all.  unfortunately i don't think there
> is a library routine that will tell us whether a string is a valid
> ip address.  parseip doesn't.
> 
> russ

Re: [9fans] $smtp dns failure

[Russ Cox]

> §5 of rfc 2821 spells out how to do the lookup.  smtpd should do that.
> if there are any huristics to apply while running the lookup, it
> would make sense to me to do as sendmail/postifix/qmail do.
> smtp is an interface to the world.  being different isn't a virtue
> in this case.

Until you explain precisely which part of rfc 2821 you claim that
smtp.c does not follow and which part of smtp.c is not following it,
I don't see a point to continuing this conversation.

The particular case that I was talking about is when dns returns
"dns: dns failure", meaning that looking up the MX record produced
an error (like maybe the DNS server choked on the request),
not that the lookup returned "there are no MX records".
My reading of the RFC is that it does not cover this case.

Russ

Re: [9fans] $smtp dns failure

[geoff]

When talking about this, it's important to distinguish between the DNS
reporting that there is no MX record for a domain and reporting that
there is a transient DNS error and thus it can't tell if there is an
MX record for a domain.

If there is no MX record, smtp should use an A or A6 record, if it
exists.  If there is a DNS error looking up the MX, smtp should stop
and not use any A or A6 record, since that may be the wrong
destination, and you can't tell until the transient DNS error passes.

Re: [9fans] $smtp dns failure

[erik quanstrom]

i'm not sure why you're taking offence.  to me it seemed clear that mxlookup
differs from the algorithm in section 5 of 2821 and what other mailers do.
the rfc doesn't say that we need to abort of the mx lookup fails.  i think this
is important because it's possible for an mx lookup to fail and an a/a6 lookup on
the same name return results.  i had quite a bit of trouble with this when using
level-5's dns last year.  in addition, a quick inspection of postfix' src/smtp/smtp_addr.c
seems to confirm that they perform a/a6 lookup on mx lookup failure.  since
other smtp agents don't notice the misconfiguration, we are left to find it.

anyway, i put the effort into pulling together my notes from last year on this
to help, not to annoy.  hopefull this information is not a further annoyance.

- erik

Re: [9fans] $smtp dns failure

[erik quanstrom]

the behaviour is not correct. rfc 2821 (section 5) says

  The lookup first attempts to locate an MX
   record associated with the name.  If a CNAME record is found instead,
   the resulting name is processed as if it were the initial name.  If
   no MX records are found, but an A RR is found, the A RR is treated as
   if it was associated with an implicit MX RR, with a preference of 0,
   pointing to that host.  If one or more MX RRs are found for a given
   name, SMTP systems MUST NOT utilize any A RRs associated with that
   name unless they are located using the MX RRs; the "implicit MX" rule
   above applies only if there are no MX records present.  If MX records
   are present, but none of them are usable, this situation MUST be
   reported as an error.

- erik

Re: [9fans] $smtp dns failure

[Paul Lalonde]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Really that should be "net!¢¢smtp!smtp"
But that's just my two cents.

On 15-Jan-07, at 11:28 AM, erik quanstrom wrote:

> '$' seems like an unfortunately choice for dial strings because
> it is easy to confuse with a shell/environment variable.
>
> perhaps a different character could be chosen?
>
> 	net!¢smtp!smtp
>
> the cost of change, unfortunately, would be high.
>
> - erik

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iD8DBQFFq9kOpJeHo/Fbu1wRAnmzAKDetJZvxOSln4Ktc/jgjLjx44JNsQCgsp3Z
coVsQkujwHhxFegXv2QJ24g=
=B2pr
-----END PGP SIGNATURE-----

Re: [9fans] $smtp dns failure

[erik quanstrom]

'$' seems like an unfortunately choice for dial strings because 
it is easy to confuse with a shell/environment variable.

perhaps a different character could be chosen?

	net!¢smtp!smtp

the cost of change, unfortunately, would be high.

- erik