Why prevent the mailing list from being indexed?

Why prevent the mailing list from being indexed?

[Stephane Chazelas]

I was surprised when a google search on the zsh mailing list
archives turned up nothing when I knew it should (site:zsh.org
ZSH_XTRACEFD) and then saw:

https://www.zsh.org/robots.txt

User-agent: *
Disallow: /cgi-bin/
Disallow: /mla/zsh-announce/
Disallow: /mla/zsh-users/
Disallow: /mla/zsh-workers/
Disallow: /pub/old/

Why?

Having google/duckduckgo searches return results from there
would be very useful, especially considering that the mailing
list is also used as an issue tracker and to some extent as a
changelog.

I know (well, I just discovered) it can be queried with
https://www.zsh.org/cgi-bin/mla/wilma_glimpse/workers?query=ZSH_XTRACEFD
but you have to know it's there and look explicitely in the ml
as opposed to generally on the web.

-- 
Stephane

Re: Why prevent the mailing list from being indexed?

[Daniel Shahaf]

Stephane Chazelas wrote on Mon, 20 May 2019 10:30 +00:00:
> I was surprised when a google search on the zsh mailing list
> archives turned up nothing when I knew it should (site:zsh.org
> ZSH_XTRACEFD) and then saw:
> 
> https://www.zsh.org/robots.txt
> 
> User-agent: *
> Disallow: /cgi-bin/
> Disallow: /mla/zsh-announce/
> Disallow: /mla/zsh-users/
> Disallow: /mla/zsh-workers/
> Disallow: /pub/old/
> 
> Why?
> 
> Having google/duckduckgo searches return results from there
> would be very useful,

Note that it only disallows the directories that contain the raw rfc822
files.  The rendered HTML indexes at /mla/workers/ aren't disallowed,
so they should be implicitly allowed, shouldn't they?

Cheers,

Daniel

> especially considering that the mailing
> list is also used as an issue tracker and to some extent as a
> changelog.
> 
> I know (well, I just discovered) it can be queried with
> https://www.zsh.org/cgi-bin/mla/wilma_glimpse/workers?query=ZSH_XTRACEFD
> but you have to know it's there and look explicitely in the ml
> as opposed to generally on the web.

Re: Why prevent the mailing list from being indexed?

[enteiser]

Mon, May 20, 2019 at 11:04:09AM +0000, Daniel Shahaf wrote:
> Stephane Chazelas wrote on Mon, 20 May 2019 10:30 +00:00:
> > I was surprised when a google search on the zsh mailing list
> > archives turned up nothing when I knew it should (site:zsh.org
> > ZSH_XTRACEFD) and then saw:
> > 
> > I know (well, I just discovered) it can be queried with
> > https://www.zsh.org/cgi-bin/mla/wilma_glimpse/workers?query=ZSH_XTRACEFD
> > but you have to know it's there and look explicitely in the ml
> > as opposed to generally on the web.

These messages are pretty new. I guess they just didn't made it into the
google index yet.

Re: Why prevent the mailing list from being indexed?

[Stephane Chazelas]

2019-05-20 11:04:09 +0000, Daniel Shahaf:
> Stephane Chazelas wrote on Mon, 20 May 2019 10:30 +00:00:
> > I was surprised when a google search on the zsh mailing list
> > archives turned up nothing when I knew it should (site:zsh.org
> > ZSH_XTRACEFD) and then saw:
> > 
> > https://www.zsh.org/robots.txt
[...]
> > Disallow: /mla/zsh-workers/
[...]
> Note that it only disallows the directories that contain the raw rfc822
> files.  The rendered HTML indexes at /mla/workers/ aren't disallowed,
> so they should be implicitly allowed, shouldn't they?
[...]

Sorry, my bad. I missed that.

I can see now they are indexed indeed.

Like enteiser@gmail.com says, it's probably just that  they're
recent.

Cheers,
Stephane