From mboxrd@z Thu Jan 1 00:00:00 1970 Mime-Version: 1.0 (Apple Message framework v753.1) In-Reply-To: References: , Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <22468C58-2225-4068-BB9F-2EA276EAAE84@fastmail.fm> Content-Transfer-Encoding: 7bit From: Ethan Grammatikidis Date: Wed, 4 May 2011 16:37:41 +0100 To: bsd@anu.homelinux.net, Fans of the OS Plan 9 from Bell Labs <9fans@9fans.net> Subject: Re: [9fans] wiki... Topicbox-Message-UUID: dc2426d6-ead6-11e9-9d60-3106f5b1d025 On 4 May 2011, at 11:40 am, Balwinder S Dheeman wrote: > On 04/26/11 12:03, Ethan Grammatikidis wrote: >> >> On 24 Apr 2011, at 9:16 am, hiro wrote: >> >>> In http://plan9.bell-labs.com/robots.txt you will find: >>> >>> User-agent: * >>> Disallow: / >> >> *facepalm* I wondered if this was the case; didn't think to check. >> Anyone have any idea why this is there? > > Very simple, since the webmaster have already allowed some bots and > disallowed everyone else ;) > > You need to read/analyze the whole robots.txt indeed. Now I've read it I can't understand why Google can't find anything under /wiki. Even if it did, that robots.txt isn't all that pleasant, blindly disallowing everyone who isn't google or msn, more or less. O.o