From mboxrd@z Thu Jan 1 00:00:00 1970 To: 9fans@9fans.net Date: Wed, 4 May 2011 11:40:18 +0000 From: Balwinder S Dheeman Message-ID: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit References: , Subject: Re: [9fans] wiki... Topicbox-Message-UUID: dbb8cb20-ead6-11e9-9d60-3106f5b1d025 On 04/26/11 12:03, Ethan Grammatikidis wrote: > > On 24 Apr 2011, at 9:16 am, hiro wrote: > >> In http://plan9.bell-labs.com/robots.txt you will find: >> >> User-agent: * >> Disallow: / > > *facepalm* I wondered if this was the case; didn't think to check. > Anyone have any idea why this is there? Very simple, since the webmaster have already allowed some bots and disallowed everyone else ;) You need to read/analyze the whole robots.txt indeed. -- Balwinder S "bdheeman" Dheeman (http://werc.homelinux.net/contact/)