From mboxrd@z Thu Jan 1 00:00:00 1970 From: dexen deVries To: Fans of the OS Plan 9 from Bell Labs <9fans@9fans.net> Date: Wed, 4 May 2011 18:11:15 +0200 User-Agent: KMail/1.13.6 (Linux/2.6.39-rc6-l23+; KDE/4.5.5; x86_64; ; ) References: <22468C58-2225-4068-BB9F-2EA276EAAE84@fastmail.fm> In-Reply-To: <22468C58-2225-4068-BB9F-2EA276EAAE84@fastmail.fm> MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <201105041811.15240.dexen.devries@gmail.com> Subject: Re: [9fans] wiki... Topicbox-Message-UUID: dcdcf72e-ead6-11e9-9d60-3106f5b1d025 On Wednesday 04 of May 2011 17:37:41 Ethan Grammatikidis wrote: > On 4 May 2011, at 11:40 am, Balwinder S Dheeman wrote: > > On 04/26/11 12:03, Ethan Grammatikidis wrote: > >> On 24 Apr 2011, at 9:16 am, hiro wrote: > >>> In http://plan9.bell-labs.com/robots.txt you will find: > >>>=20 > >>> User-agent: * > >>> Disallow: / > >>=20 > >> *facepalm* I wondered if this was the case; didn't think to check. > >> Anyone have any idea why this is there? > >=20 > > Very simple, since the webmaster have already allowed some bots and > > disallowed everyone else ;) > >=20 > > You need to read/analyze the whole robots.txt indeed. >=20 > Now I've read it I can't understand why Google can't find anything > under /wiki. Even if it did, that robots.txt isn't all that pleasant, > blindly disallowing everyone who isn't google or msn, more or less. O.o I believe we need an ``Allow: /'' below the long list of `Disallows' in the= =20 User-agent: Googlebot, User-agent: msnbot section. Otherwise, only the fina= l=20 ``Disallow: /'' matches, and in effecet, every robot is cut off. Or, better, just let any robots crawl the site. Web isn't only about google= =20 and msn anymore ;-) (*cough* http://duckduckgo.com/ *cough*) =2D-=20 dexen deVries [[[=E2=86=93][=E2=86=92]]] ``In other news, STFU and hack.'' mahmud, in response to Erann Gat's ``How I lost my faith in Lisp'' http://news.ycombinator.com/item?id=3D2308816