From mboxrd@z Thu Jan 1 00:00:00 1970 From: erik quanstrom Date: Thu, 5 May 2011 08:38:20 -0400 To: 9fans@9fans.net Message-ID: <4f55e9c67f685d98789b646fc247763f@brasstown.quanstro.net> In-Reply-To: <7sl898xc9s.ln2@news.homelinux.net> References: <22468C58-2225-4068-BB9F-2EA276EAAE84@fastmail.fm> <7sl898xc9s.ln2@news.homelinux.net> MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Subject: Re: [9fans] wiki... Topicbox-Message-UUID: dd80127e-ead6-11e9-9d60-3106f5b1d025 > I maintain anu.homelinux.net and werc.homelinux.net sites, mine > robots.txt files are quite simple and I never ever facing these servers > being chocked by any such bot. i think it's a mistake to generalized from a sample size of 1 or 2 to the whole world. give 'em a break. - erik