From mboxrd@z Thu Jan 1 00:00:00 1970 To: 9fans@9fans.net Date: Thu, 5 May 2011 09:54:19 +0000 From: Balwinder S Dheeman Message-ID: <7sl898xc9s.ln2@news.homelinux.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit References: , <22468C58-2225-4068-BB9F-2EA276EAAE84@fastmail.fm> Subject: Re: [9fans] wiki... Topicbox-Message-UUID: dd585e78-ead6-11e9-9d60-3106f5b1d025 On 05/04/2011 09:09 PM, Ethan Grammatikidis wrote: > > On 4 May 2011, at 11:40 am, Balwinder S Dheeman wrote: > >> On 04/26/11 12:03, Ethan Grammatikidis wrote: >>> >>> On 24 Apr 2011, at 9:16 am, hiro wrote: >>> >>>> In http://plan9.bell-labs.com/robots.txt you will find: >>>> >>>> User-agent: * >>>> Disallow: / >>> >>> *facepalm* I wondered if this was the case; didn't think to check. >>> Anyone have any idea why this is there? >> >> Very simple, since the webmaster have already allowed some bots and >> disallowed everyone else ;) >> >> You need to read/analyze the whole robots.txt indeed. > > Now I've read it I can't understand why Google can't find anything under > /wiki. Even if it did, that robots.txt isn't all that pleasant, blindly > disallowing everyone who isn't google or msn, more or less. O.o May be either the robots.txt is incorrect or Google and MSN/Bing are interpreting it in wrong manner. I maintain anu.homelinux.net and werc.homelinux.net sites, mine robots.txt files are quite simple and I never ever facing these servers being chocked by any such bot. -- Balwinder S "bdheeman" Dheeman (http://werc.homelinux.net/contact/)