[9fans] lsub.org

[9fans] lsub.org

[ron minnich]

gone?

ron

Re: [9fans] lsub.org

[Gorka Guardiola]

Problems with the dns and everyone out of town. Will be back as soon
as possible :-).

On 8/12/07, ron minnich <rminnich@gmail.com> wrote:
> gone?
>
> ron
>


-- 
- curiosity sKilled the cat

Re: [9fans] lsub.org

[erik quanstrom]

assuming things are broken without external help ....

i've been having trouble with dns infinitely extending
the life of queries when a "srvfail" is returned by an authoratitive
server.  eventually one query to a broken ns will hold up all the threads available
on the server.  this happends a lot on reverse lookups.  i fire this script
every 10 minutes to help ease the pain until i have the time figure out exactly
what's going wrong.

- erik

#!/bin/rc
rfork en
mailuser=guywhogetstocheckonthisstuff
fflag=0
nl='
'
fn usage{
	echo 'usage: restartdns [-f]' >[1=2]
	exit usage
}

fn why{
	if(! ~ $#nbroken 0)
		echo getting mediæval on $#nbroken broken dns processes.
	if not{
		echo getting mediæval on $#nwait deadlocked dns processes.
		for(i in $nwait)
			echo $i
	}
}

for(i)switch($i){
case -f
	fflag=1
case *
	usage
}

if(~ $fflag 0){
	nbroken=`{ps -a | grep dns | grep Broken}
	ifs=$nl nwait=`{ps -a |sed -n 's/.* +dns \[query lock wait for(.*)\]/\1/gp' | sort | uniq -c | awk '$1>2'}

	if(~ $#nbroken 0 && ~ $#nwait 0)
		exit 'none broken'
	why
	if(~ $service rx)
		{date; echo; why; echo; ps -a | grep dns}| mail $guywhogetstocheckonthisstuff
}

slay dns | rc
ndb/dns -s
ndb/dns -Rrsx /net.alt -f /lib/ndb/external

Re: [9fans] lsub.org

[erik quanstrom]

need to sed the brain this morning

On Mon Aug 13 07:57:00 EDT 2007, quanstro@quanstro.net wrote:
> assuming things are broken without external help ....
> 

s/external/internal/

- erik

Re: [9fans] lsub.org

[Francisco J Ballesteros]

That´s it. lsub.org is gone.
I think our dns is out of procs, and nobody is in town. I don´t know
if I´ll have
to go back to madrid, or just wait until next Sat, when I had the plan
to return.

Thanks a lot for the script, that´ll be the first thing I´ll install
upon return.
(btw, nemo.mbox@gmail.com is still a way to reach me, should anyone have to).


On 8/13/07, erik quanstrom <quanstro@quanstro.net> wrote:
> assuming things are broken without external help ....
>
> i've been having trouble with dns infinitely extending
> the life of queries when a "srvfail" is returned by an authoratitive
> server.  eventually one query to a broken ns will hold up all the threads available
> on the server.  this happends a lot on reverse lookups.  i fire this script
> every 10 minutes to help ease the pain until i have the time figure out exactly
> what's going wrong.
>
> - erik
>
> #!/bin/rc
> rfork en
> mailuser=guywhogetstocheckonthisstuff
> fflag=0
> nl='
> '
> fn usage{
>         echo 'usage: restartdns [-f]' >[1=2]
>         exit usage
> }
>
> fn why{
>         if(! ~ $#nbroken 0)
>                 echo getting mediæval on $#nbroken broken dns processes.
>         if not{
>                 echo getting mediæval on $#nwait deadlocked dns processes.
>                 for(i in $nwait)
>                         echo $i
>         }
> }
>
> for(i)switch($i){
> case -f
>         fflag=1
> case *
>         usage
> }
>
> if(~ $fflag 0){
>         nbroken=`{ps -a | grep dns | grep Broken}
>         ifs=$nl nwait=`{ps -a |sed -n 's/.* +dns \[query lock wait for(.*)\]/\1/gp' | sort | uniq -c | awk '$1>2'}
>
>         if(~ $#nbroken 0 && ~ $#nwait 0)
>                 exit 'none broken'
>         why
>         if(~ $service rx)
>                 {date; echo; why; echo; ps -a | grep dns}| mail $guywhogetstocheckonthisstuff
> }
>
> slay dns | rc
> ndb/dns -s
> ndb/dns -Rrsx /net.alt -f /lib/ndb/external
>

Re: [9fans] lsub.org

[Francisco J Ballesteros]

lsub.org is back again. Thanks to help from someone in the building.


On 8/13/07, Francisco J Ballesteros <nemo@lsub.org> wrote:
> That´s it. lsub.org is gone.
> I think our dns is out of procs, and nobody is in town. I don´t know
> if I´ll have
> to go back to madrid, or just wait until next Sat, when I had the plan
> to return.
>
> Thanks a lot for the script, that´ll be the first thing I´ll install
> upon return.
> (btw, nemo.mbox@gmail.com is still a way to reach me, should anyone have to).
>
>
> On 8/13/07, erik quanstrom <quanstro@quanstro.net> wrote:
> > assuming things are broken without external help ....
> >
> > i've been having trouble with dns infinitely extending
> > the life of queries when a "srvfail" is returned by an authoratitive
> > server.  eventually one query to a broken ns will hold up all the threads available
> > on the server.  this happends a lot on reverse lookups.  i fire this script
> > every 10 minutes to help ease the pain until i have the time figure out exactly
> > what's going wrong.
> >
> > - erik
> >
> > #!/bin/rc
> > rfork en
> > mailuser=guywhogetstocheckonthisstuff
> > fflag=0
> > nl='
> > '
> > fn usage{
> >         echo 'usage: restartdns [-f]' >[1=2]
> >         exit usage
> > }
> >
> > fn why{
> >         if(! ~ $#nbroken 0)
> >                 echo getting mediæval on $#nbroken broken dns processes.
> >         if not{
> >                 echo getting mediæval on $#nwait deadlocked dns processes.
> >                 for(i in $nwait)
> >                         echo $i
> >         }
> > }
> >
> > for(i)switch($i){
> > case -f
> >         fflag=1
> > case *
> >         usage
> > }
> >
> > if(~ $fflag 0){
> >         nbroken=`{ps -a | grep dns | grep Broken}
> >         ifs=$nl nwait=`{ps -a |sed -n 's/.* +dns \[query lock wait for(.*)\]/\1/gp' | sort | uniq -c | awk '$1>2'}
> >
> >         if(~ $#nbroken 0 && ~ $#nwait 0)
> >                 exit 'none broken'
> >         why
> >         if(~ $service rx)
> >                 {date; echo; why; echo; ps -a | grep dns}| mail $guywhogetstocheckonthisstuff
> > }
> >
> > slay dns | rc
> > ndb/dns -s
> > ndb/dns -Rrsx /net.alt -f /lib/ndb/external
> >
>

Re: [9fans] lsub.org

[geoff]

We've been having good luck with the dns currently on sources.  I
don't think I have seen the srvfail problem.  The query waits on
reverse lookups seem to have been a symptom of local ndb
misconfiguration, though it would be better if dns coped more sensibly
with that.  As I recall, the nameserver(s) for the reverse domain need
to have the relevant ptr records; pointing the reverse domain at the
wrong nameservers (implicitly or explicitly) seems to cause confusion.

The main outstanding bug that I'm aware of is that aging resource
records seems to corrupt some data structure and dns never recovers.
There are two new control messages intended to help debug this:
"target N" and "age" (see dns.c).  But if one doesn't age resource
records, dns can grow without bound.

Re: [9fans] lsub.org

[erik quanstrom]

On Mon Aug 13 14:31:57 EDT 2007, geoff@plan9.bell-labs.com wrote:
> We've been having good luck with the dns currently on sources.  I
> don't think I have seen the srvfail problem.  The query waits on
> reverse lookups seem to have been a symptom of local ndb
> misconfiguration, though it would be better if dns coped more sensibly
> with that.  As I recall, the nameserver(s) for the reverse domain need
> to have the relevant ptr records; pointing the reverse domain at the
> wrong nameservers (implicitly or explicitly) seems to cause confusion.
> 
> The main outstanding bug that I'm aware of is that aging resource
> records seems to corrupt some data structure and dns never recovers.
> There are two new control messages intended to help debug this:
> "target N" and "age" (see dns.c).  But if one doesn't age resource
> records, dns can grow without bound.

this is a problem i see with recursive nameservers doing lookups for
email validataion, so it's not a question of local server misconfiguration.
there's nothing i can do if arin or whoever has the wrong information.

- erik

Re: [9fans] lsub.org

[erik quanstrom]

> We've been having good luck with the dns currently on sources.  I
> don't think I have seen the srvfail problem.  The query waits on
> reverse lookups seem to have been a symptom of local ndb
> misconfiguration, though it would be better if dns coped more sensibly
> with that.  As I recall, the nameserver(s) for the reverse domain need
> to have the relevant ptr records; pointing the reverse domain at the
> wrong nameservers (implicitly or explicitly) seems to cause confusion.

could you explain this senerio more?

when i see queries hanging, it's almost always on recursive reverse lookups
for email validataion.  some external server will return srvfail but the query
will be restarted from the dns root ad inf.  this results on other queries to the
same zone to wait for the query lock forever.  since spam often comes in
bunches from the same domains, this often uses up all the available responding
threads.

what could be wrong in the local ndb file for this to be happening?

- erik

Re: [9fans] lsub.org

[geoff]

I can't remember more details of the reverse-lookup query-wait
problem; sorry.

Re: [9fans] lsub.org

[erik quanstrom]

on the dns front, i've found that some spam senders are
arranging things so that the guys doing reverse-lookup
validataion will get 192.168 or 10. addresses.  for some reason
arin doesn't return an address for a query on 10.in-addr.arpa
or 168.192.in-addr.arpa, so dns will loop from the top and never
time out.

this doesn't fix the problem, but it will stop these kinds of queries in
their tracks.  add to /lib/ndb/$myrecursiveserver:

# 
# spam defense.  unfortunately, arin doesn't give negative
# rcodes for these non-routable addresses.  we'll do it for
# them
#
dom=168.192.in-addr.arpa soa=
	refresh=3600 ttl=3600
	ns=ns1.MY.DOM
	ns=ns2.MY.DOM

dom=10.in-addr.arpa soa=
	refresh=3600 ttl=3600
	ns=ns1.MY.DOM
	ns=ns2.MY.DOM

- erik

Re: [9fans] lsub.org

[Artem Letko]

something like that.

see http://www.ietf.org/ids.by.wg/dnsop.html

in particular http://www.ietf.org/internet-drafts/draft-ietf-dnsop-default-local-zones-02.txt

-art

On 8/14/07, erik quanstrom <quanstro@coraid.com> wrote:
> on the dns front, i've found that some spam senders are
> arranging things so that the guys doing reverse-lookup
> validataion will get 192.168 or 10. addresses.  for some reason
> arin doesn't return an address for a query on 10.in-addr.arpa
> or 168.192.in-addr.arpa, so dns will loop from the top and never
> time out.
>
> this doesn't fix the problem, but it will stop these kinds of queries in
> their tracks.  add to /lib/ndb/$myrecursiveserver:
>
> #
> # spam defense.  unfortunately, arin doesn't give negative
> # rcodes for these non-routable addresses.  we'll do it for
> # them
> #
> dom=168.192.in-addr.arpa soa=
>         refresh=3600 ttl=3600
>         ns=ns1.MY.DOM
>         ns=ns2.MY.DOM
>
> dom=10.in-addr.arpa soa=
>         refresh=3600 ttl=3600
>         ns=ns1.MY.DOM
>         ns=ns2.MY.DOM
>
> - erik
>

Re: [9fans] lsub.org

[erik quanstrom]

> something like that.
> 
> see http://www.ietf.org/ids.by.wg/dnsop.html
> 
> in particular http://www.ietf.org/internet-drafts/draft-ietf-dnsop-default-local-zones-02.txt
> 
> -art

are you saying you've see the same email trick with addresses
other than 10/8 or 192.168/16?

- erik

Re: [9fans] lsub.org

[Artem Letko]

no, just pointing out relevant ietf draft.

-art

On 8/14/07, erik quanstrom <quanstro@quanstro.net> wrote:
> > something like that.
> >
> > see http://www.ietf.org/ids.by.wg/dnsop.html
> >
> > in particular http://www.ietf.org/internet-drafts/draft-ietf-dnsop-default-local-zones-02.txt
> >
> > -art
>
> are you saying you've see the same email trick with addresses
> other than 10/8 or 192.168/16?
>
> - erik
>