Re: [9fans] wiki...

[dexen deVries <dexen.devries@gmail.com>]

On Wednesday 04 of May 2011 17:37:41 Ethan Grammatikidis wrote:
> On 4 May 2011, at 11:40 am, Balwinder S Dheeman wrote:
> > On 04/26/11 12:03, Ethan Grammatikidis wrote:
> >> On 24 Apr 2011, at 9:16 am, hiro wrote:
> >>> In http://plan9.bell-labs.com/robots.txt you will find:
> >>> 
> >>> User-agent: *
> >>> Disallow: /
> >> 
> >> *facepalm* I wondered if this was the case; didn't think to check.
> >> Anyone have any idea why this is there?
> > 
> > Very simple, since the webmaster have already allowed some bots and
> > disallowed everyone else ;)
> > 
> > You need to read/analyze the whole robots.txt indeed.
> 
> Now I've read it I can't understand why Google can't find anything
> under /wiki. Even if it did, that robots.txt isn't all that pleasant,
> blindly disallowing everyone who isn't google or msn, more or less. O.o



I believe we need an ``Allow: /'' below the long list of `Disallows' in the 
User-agent: Googlebot, User-agent: msnbot section. Otherwise, only the final 
``Disallow: /'' matches, and in effecet, every robot is cut off.


Or, better, just let any robots crawl the site. Web isn't only about google 
and msn anymore ;-)

(*cough* http://duckduckgo.com/ *cough*)

-- 
dexen deVries

[[[↓][→]]]

``In other news, STFU and hack.''
mahmud, in response to Erann Gat's ``How I lost my faith in Lisp''
http://news.ycombinator.com/item?id=2308816