zsh-users
 help / Atom feed
* Why prevent the mailing list from being indexed?
@ 2019-05-20 10:28 Stephane Chazelas
  2019-05-20 11:04 ` Daniel Shahaf
       [not found] ` <a9448482-d248-4e9d-9ffe-318d51c781fb__45528.5614518047$1558350374$gmane$org@www.fastmail.com>
  0 siblings, 2 replies; 4+ messages in thread
From: Stephane Chazelas @ 2019-05-20 10:28 UTC (permalink / raw)
  To: Zsh Users List

I was surprised when a google search on the zsh mailing list
archives turned up nothing when I knew it should (site:zsh.org
ZSH_XTRACEFD) and then saw:

https://www.zsh.org/robots.txt

User-agent: *
Disallow: /cgi-bin/
Disallow: /mla/zsh-announce/
Disallow: /mla/zsh-users/
Disallow: /mla/zsh-workers/
Disallow: /pub/old/

Why?

Having google/duckduckgo searches return results from there
would be very useful, especially considering that the mailing
list is also used as an issue tracker and to some extent as a
changelog.

I know (well, I just discovered) it can be queried with
https://www.zsh.org/cgi-bin/mla/wilma_glimpse/workers?query=ZSH_XTRACEFD
but you have to know it's there and look explicitely in the ml
as opposed to generally on the web.

-- 
Stephane

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Why prevent the mailing list from being indexed?
  2019-05-20 10:28 Why prevent the mailing list from being indexed? Stephane Chazelas
@ 2019-05-20 11:04 ` Daniel Shahaf
  2019-05-20 11:57   ` enteiser
       [not found] ` <a9448482-d248-4e9d-9ffe-318d51c781fb__45528.5614518047$1558350374$gmane$org@www.fastmail.com>
  1 sibling, 1 reply; 4+ messages in thread
From: Daniel Shahaf @ 2019-05-20 11:04 UTC (permalink / raw)
  To: zsh-users

Stephane Chazelas wrote on Mon, 20 May 2019 10:30 +00:00:
> I was surprised when a google search on the zsh mailing list
> archives turned up nothing when I knew it should (site:zsh.org
> ZSH_XTRACEFD) and then saw:
> 
> https://www.zsh.org/robots.txt
> 
> User-agent: *
> Disallow: /cgi-bin/
> Disallow: /mla/zsh-announce/
> Disallow: /mla/zsh-users/
> Disallow: /mla/zsh-workers/
> Disallow: /pub/old/
> 
> Why?
> 
> Having google/duckduckgo searches return results from there
> would be very useful,

Note that it only disallows the directories that contain the raw rfc822
files.  The rendered HTML indexes at /mla/workers/ aren't disallowed,
so they should be implicitly allowed, shouldn't they?

Cheers,

Daniel

> especially considering that the mailing
> list is also used as an issue tracker and to some extent as a
> changelog.
> 
> I know (well, I just discovered) it can be queried with
> https://www.zsh.org/cgi-bin/mla/wilma_glimpse/workers?query=ZSH_XTRACEFD
> but you have to know it's there and look explicitely in the ml
> as opposed to generally on the web.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Why prevent the mailing list from being indexed?
  2019-05-20 11:04 ` Daniel Shahaf
@ 2019-05-20 11:57   ` enteiser
  0 siblings, 0 replies; 4+ messages in thread
From: enteiser @ 2019-05-20 11:57 UTC (permalink / raw)
  To: zsh-users

Mon, May 20, 2019 at 11:04:09AM +0000, Daniel Shahaf wrote:
> Stephane Chazelas wrote on Mon, 20 May 2019 10:30 +00:00:
> > I was surprised when a google search on the zsh mailing list
> > archives turned up nothing when I knew it should (site:zsh.org
> > ZSH_XTRACEFD) and then saw:
> > 
> > I know (well, I just discovered) it can be queried with
> > https://www.zsh.org/cgi-bin/mla/wilma_glimpse/workers?query=ZSH_XTRACEFD
> > but you have to know it's there and look explicitely in the ml
> > as opposed to generally on the web.

These messages are pretty new. I guess they just didn't made it into the
google index yet.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Why prevent the mailing list from being indexed?
       [not found] ` <a9448482-d248-4e9d-9ffe-318d51c781fb__45528.5614518047$1558350374$gmane$org@www.fastmail.com>
@ 2019-05-20 14:28   ` Stephane Chazelas
  0 siblings, 0 replies; 4+ messages in thread
From: Stephane Chazelas @ 2019-05-20 14:28 UTC (permalink / raw)
  To: Daniel Shahaf; +Cc: zsh-users

2019-05-20 11:04:09 +0000, Daniel Shahaf:
> Stephane Chazelas wrote on Mon, 20 May 2019 10:30 +00:00:
> > I was surprised when a google search on the zsh mailing list
> > archives turned up nothing when I knew it should (site:zsh.org
> > ZSH_XTRACEFD) and then saw:
> > 
> > https://www.zsh.org/robots.txt
[...]
> > Disallow: /mla/zsh-workers/
[...]
> Note that it only disallows the directories that contain the raw rfc822
> files.  The rendered HTML indexes at /mla/workers/ aren't disallowed,
> so they should be implicitly allowed, shouldn't they?
[...]

Sorry, my bad. I missed that.

I can see now they are indexed indeed.

Like enteiser@gmail.com says, it's probably just that  they're
recent.

Cheers,
Stephane

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, back to index

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-20 10:28 Why prevent the mailing list from being indexed? Stephane Chazelas
2019-05-20 11:04 ` Daniel Shahaf
2019-05-20 11:57   ` enteiser
     [not found] ` <a9448482-d248-4e9d-9ffe-318d51c781fb__45528.5614518047$1558350374$gmane$org@www.fastmail.com>
2019-05-20 14:28   ` Stephane Chazelas

zsh-users

Archives are clonable: git clone --mirror http://inbox.vuxu.org/zsh-users

Newsgroup available over NNTP:
	nntp://inbox.vuxu.org/vuxu.archive.zsh.users


AGPL code for this site: git clone https://public-inbox.org/ public-inbox