[9fans] source browsing via http is back

[9fans] source browsing via http is back

[geoff]

I've turned it back on and will watch to see if our web server gets
swamped by it.  This interface should not be used to mirror the
contents of sources.

Re: [9fans] source browsing via http is back

[Bruce Ellis]

Thank you geoff. You are one of the good guys. And I've never heard you whine.

Now, back to work for brucee and Tiger.

brucee

On Wed, Feb 11, 2009 at 5:49 AM,  <geoff@plan9.bell-labs.com> wrote:
> I've turned it back on and will watch to see if our web server gets
> swamped by it.  This interface should not be used to mirror the
> contents of sources.
>
>

Re: [9fans] source browsing via http is back

[John Barham]

> I've turned it back on and will watch to see if our web server gets
> swamped by it.  This interface should not be used to mirror the
> contents of sources.

What interface should be used to used to mirror sources?  9fs?
replica?  More specifically, if I wanted to set up my own HTTP mirror
of sources, what is the labs approved way to stay in sync?

Re: [9fans] source browsing via http is back

[ron minnich]

On Tue, Feb 10, 2009 at 1:10 PM, John Barham <jbarham@gmail.com> wrote:
>> I've turned it back on and will watch to see if our web server gets
>> swamped by it.  This interface should not be used to mirror the
>> contents of sources.
>
> What interface should be used to used to mirror sources?  9fs?
> replica?  More specifically, if I wanted to set up my own HTTP mirror
> of sources, what is the labs approved way to stay in sync?
>
>

since 9fs never stopped working, why could you not just export (with
httpd) from your machine a -C mount of sources? then the robots hammer
you but since you're cached, you don't hammer sources?

ron

Re: [9fans] source browsing via http is back

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 1606 bytes --]

On Tue, Feb 10, 2009 at 01:15:11PM -0800, ron minnich wrote:
> On Tue, Feb 10, 2009 at 1:10 PM, John Barham <jbarham@gmail.com> wrote:
> >> I've turned it back on and will watch to see if our web server gets
> >> swamped by it.  This interface should not be used to mirror the
> >> contents of sources.
> >
> > What interface should be used to used to mirror sources?  9fs?
> > replica?  More specifically, if I wanted to set up my own HTTP mirror
> > of sources, what is the labs approved way to stay in sync?
> >
> >
> 
> since 9fs never stopped working, why could you not just export (with
> httpd) from your machine a -C mount of sources? then the robots hammer
> you but since you're cached, you don't hammer sources?

Sorry to rain on your parade as it were, but occasionally 9fs to sources
does, in fact, stop working.  See my and others' posts to 9fans about
sources and especially sources' venti being down.

I again propose that sources should be mirrorable via venti (and venti/copy
-f); the nightly snapshots would be walked with "auth/none vac" into a
publically readable venti (venti/ro proxy) and the scores published (and
signed).  In order to make this more palatable, it may be worth developing a
venti proxy that attempts to pull blocks from closer ventis rather than
further ones and to make sources' public venti artificially further away
(induce a delay of half a second or something per Tread).  I've some
tentative writeup on https://wiki.ietfng.org/pub/Plan9/VentiMirror and even
some code flying locally but nothing about which to be proud.

--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] source browsing via http is back

[erik quanstrom]

> I again propose that sources should be mirrorable via venti (and venti/copy
> -f); the nightly snapshots would be walked with "auth/none vac" into a
> publically readable venti (venti/ro proxy) and the scores published (and
> signed).  In order to make this more palatable, it may be worth developing a
> venti proxy that attempts to pull blocks from closer ventis rather than
> further ones and to make sources' public venti artificially further away
> (induce a delay of half a second or something per Tread).  I've some
> tentative writeup on https://wiki.ietfng.org/pub/Plan9/VentiMirror and even
> some code flying locally but nothing about which to be proud.

i'm still not following why replica won't work?  getting in underneath
the fs seems to require some extra justification and it seems to require
some very low-level modifications.  and yet the file interface provides
what i think one would need for such a project.  what is wrong with
replica that can't be reasonably fixed?

- erik

Re: [9fans] source browsing via http is back

[Roman V. Shaposhnik]

On Tue, 2009-02-10 at 16:32 -0500, erik quanstrom wrote:
> > I again propose that sources should be mirrorable via venti (and venti/copy
> > -f); the nightly snapshots would be walked with "auth/none vac" into a
> > publically readable venti (venti/ro proxy) and the scores published (and
> > signed).  In order to make this more palatable, it may be worth developing a
> > venti proxy that attempts to pull blocks from closer ventis rather than
> > further ones and to make sources' public venti artificially further away
> > (induce a delay of half a second or something per Tread).  I've some
> > tentative writeup on https://wiki.ietfng.org/pub/Plan9/VentiMirror and even
> > some code flying locally but nothing about which to be proud.
>
> i'm still not following why replica won't work?  getting in underneath
> the fs seems to require some extra justification and it seems to require
> some very low-level modifications.  and yet the file interface provides
> what i think one would need for such a project.  what is wrong with
> replica that can't be reasonably fixed?

since replica requires some (albeit automatic) periodic work on the
server end it means that there's one more thing for bell lab folks
to care about and maintain. that said, a brand new Venti proxy or
what not will probably be even worse for them :-(

Thanks,
Roman.

Re: [9fans] source browsing via http is back

[erik quanstrom]

> > i'm still not following why replica won't work?  getting in underneath
> > the fs seems to require some extra justification and it seems to require
> > some very low-level modifications.  and yet the file interface provides
> > what i think one would need for such a project.  what is wrong with
> > replica that can't be reasonably fixed?
>
> since replica requires some (albeit automatic) periodic work on the
> server end it means that there's one more thing for bell lab folks
> to care about and maintain. that said, a brand new Venti proxy or
> what not will probably be even worse for them :-(

the replica logs are already generated and already required
for existing functionality — pull — to work.

- erik

Re: [9fans] source browsing via http is back

[Roman V. Shaposhnik]

On Tue, 2009-02-10 at 16:55 -0500, erik quanstrom wrote:
> > > i'm still not following why replica won't work?  getting in underneath
> > > the fs seems to require some extra justification and it seems to require
> > > some very low-level modifications.  and yet the file interface provides
> > > what i think one would need for such a project.  what is wrong with
> > > replica that can't be reasonably fixed?
> > 
> > since replica requires some (albeit automatic) periodic work on the
> > server end it means that there's one more thing for bell lab folks
> > to care about and maintain. that said, a brand new Venti proxy or
> > what not will probably be even worse for them :-(
> 
> the replica logs are already generated and already required
> for existing functionality — pull — to work.

What do you mean "they are already generated"? The script(s) try to
generate them. When they fail they need a manual intervention.
That was my entire point.

Thanks,
Roman.

Re: [9fans] source browsing via http is back

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

On Tue, Feb 10, 2009 at 01:51:47PM -0800, Roman V. Shaposhnik wrote:
> since replica requires some (albeit automatic) periodic work on the
> server end it means that there's one more thing for bell lab folks
> to care about and maintain. that said, a brand new Venti proxy or
> what not will probably be even worse for them :-(

Under the proposed scheme, clients and maybe mirrors, not sources, run the
new proxy.  Sources runs only a venti and venti/ro, unmodified, to allow
clients and mirrors to fetch blocks, and auth/none vac.  Is this unreasonable?

--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] source browsing via http is back

[Roman V. Shaposhnik]

On Tue, 2009-02-10 at 17:13 -0500, Nathaniel W Filardo wrote:
> On Tue, Feb 10, 2009 at 01:51:47PM -0800, Roman V. Shaposhnik wrote:
> > since replica requires some (albeit automatic) periodic work on the
> > server end it means that there's one more thing for bell lab folks
> > to care about and maintain. that said, a brand new Venti proxy or
> > what not will probably be even worse for them :-(
>
> Under the proposed scheme, clients and maybe mirrors, not sources, run the
> new proxy.  Sources runs only a venti and venti/ro, unmodified, to allow
> clients and mirrors to fetch blocks, and auth/none vac.  Is this unreasonable?

It would be completely reasonable (and I wager *less* work for bell labs
folks than keeping an eye on replica) if that venti was only hosting
blocks coming from sources. As far as I know that's not the case.

Thanks,
Roman.

Re: [9fans] source browsing via http is back

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

On Tue, Feb 10, 2009 at 04:32:18PM -0500, erik quanstrom wrote:
> i'm still not following why replica won't work?  getting in underneath
> the fs seems to require some extra justification and it seems to require
> some very low-level modifications.  and yet the file interface provides
> what i think one would need for such a project.  what is wrong with
> replica that can't be reasonably fixed?

I wish to enable mirrors of sourcesdump as well.  Replica would do lots of
extra work and transfer, vs. venti/copy -f, yes?

--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] source browsing via http is back

[erik quanstrom]

> On Tue, Feb 10, 2009 at 04:32:18PM -0500, erik quanstrom wrote:
> > i'm still not following why replica won't work?  getting in underneath
> > the fs seems to require some extra justification and it seems to require
> > some very low-level modifications.  and yet the file interface provides
> > what i think one would need for such a project.  what is wrong with
> > replica that can't be reasonably fixed?
>
> I wish to enable mirrors of sourcesdump as well.  Replica would do lots of
> extra work and transfer, vs. venti/copy -f, yes?

what extra work would that be, and if there is extra work, could you explain
why this would be a problem?

- erik

Re: [9fans] source browsing via http is back

[Roman V. Shaposhnik]

On Tue, 2009-02-10 at 17:10 -0500, erik quanstrom wrote:
> > On Tue, Feb 10, 2009 at 04:32:18PM -0500, erik quanstrom wrote:
> > > i'm still not following why replica won't work?  getting in underneath
> > > the fs seems to require some extra justification and it seems to require
> > > some very low-level modifications.  and yet the file interface provides
> > > what i think one would need for such a project.  what is wrong with
> > > replica that can't be reasonably fixed?
> >
> > I wish to enable mirrors of sourcesdump as well.  Replica would do lots of
> > extra work and transfer, vs. venti/copy -f, yes?
>
> what extra work would that be, and if there is extra work, could you explain
> why this would be a problem?

Correct me if I'm wrong, but my understanding of how pull works is
that it pulls files in their entirety (IOW: cp /n/sources/... /)
which means that shared blocks get copied as many times as there
are files sharing them.

Thanks,
Roman.

Re: [9fans] source browsing via http is back

[erik quanstrom]

> > what extra work would that be, and if there is extra work, could you explain
> > why this would be a problem?
>
> Correct me if I'm wrong, but my understanding of how pull works is
> that it pulls files in their entirety (IOW: cp /n/sources/... /)
> which means that shared blocks get copied as many times as there
> are files sharing them.

i would be very suprised if a single copy of sources had many shared
blocks.  furthermore, as is made quite plain by upas, a change early
in a file almost always changes every subsequent block in the file.
there are some corner cases for block-sized or same sized adds/deletes.
but nontheless, the vast majority of changes will change a good deal
of of the blocks comprising the file.

what leads you to beleve that that amount of sharing will be
significant?

- erik

Re: [9fans] source browsing via http is back

[Roman V. Shaposhnik]

On Tue, 2009-02-10 at 17:28 -0500, erik quanstrom wrote:
> what leads you to beleve that that amount of sharing will be
> significant?

Just a hunch so far. I don't have hard data to prove anything.
On the other hand, I'd be surprised if massive updates (not pulling
in a couple of months) didn't benefit from the sharing.

Thanks,
Roman.

Re: [9fans] source browsing via http is back

[Bruce Ellis]

get over it. it works fine on my found machine. and my $300 eeePC.
what exactly doesn't work?

brucee

On Wed, Feb 11, 2009 at 9:45 AM, Roman V. Shaposhnik <rvs@sun.com> wrote:
> On Tue, 2009-02-10 at 17:28 -0500, erik quanstrom wrote:
>> what leads you to beleve that that amount of sharing will be
>> significant?
>
> Just a hunch so far. I don't have hard data to prove anything.
> On the other hand, I'd be surprised if massive updates (not pulling
> in a couple of months) didn't benefit from the sharing.
>
> Thanks,
> Roman.
>
>
>

Re: [9fans] source browsing via http is back

[Roman V. Shaposhnik]

On Wed, 2009-02-11 at 11:22 +1100, Bruce Ellis wrote:
> get over it. it works fine on my found machine. and my $300 eeePC.
> what exactly doesn't work?

no argument there. my automechanic still uses MS DOS 5.0. works
great for him. i tried to tell him about plan9, but strangely
enough he also told me to get over it.

Thanks,
Roman.

p.s. oh, and he's charges the least for the most reasonable
service in the area. think coincidence? i think not!

Re: [9fans] source browsing via http is back

[Bruce Ellis]

i have no problem with pragmatism.

brucee

On Wed, Feb 11, 2009 at 11:28 AM, Roman V. Shaposhnik <rvs@sun.com> wrote:
> On Wed, 2009-02-11 at 11:22 +1100, Bruce Ellis wrote:
>> get over it. it works fine on my found machine. and my $300 eeePC.
>> what exactly doesn't work?
>
> no argument there. my automechanic still uses MS DOS 5.0. works
> great for him. i tried to tell him about plan9, but strangely
> enough he also told me to get over it.
>
> Thanks,
> Roman.
>
> p.s. oh, and he's charges the least for the most reasonable
> service in the area. think coincidence? i think not!
>
>
>

Re: [9fans] source browsing via http is back

[Akshat Kumar]

> it works fine on my found machine. and my $300 eeePC.

9vx (built on another computer, and off of an external HDD) works just
fine in eeePC.  But what are you using for a Plan 9 environment on
that thing?


in case your milkshake's better than mine
ak

[9fans] Plan 9 source history (was: Re: source browsing via http is back)

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 2091 bytes --]

On Tue, Feb 10, 2009 at 02:45:43PM -0800, Roman V. Shaposhnik wrote:
> On Tue, 2009-02-10 at 17:28 -0500, erik quanstrom wrote:
> > what leads you to beleve that that amount of sharing will be
> > significant?
> 
> Just a hunch so far. I don't have hard data to prove anything.
> On the other hand, I'd be surprised if massive updates (not pulling
> in a couple of months) didn't benefit from the sharing.
> 
> Thanks,
> Roman.

I have mirrored, with vac -f, every sources dump from 2002 to
yesterday with 
      -e acme/acid/386 -e acme/acid/alpha -e acme/acid/arm \
      -e acme/acid/mips -e acme/acid/power -e acme/bin/386 \
      -e acme/bin/alpha -e acme/bin/arm -e acme/bin/mips \
      -e acme/bin/power -e acme/mail/386 -e acme/mail/alpha \
      -e acme/mail/arm -e acme/mail/mips -e acme/mail/power \
      -e sys/man/vol1.ps -e sys/man/vol1.ps.gz -e sys/man/vol1.pdf \
      LICENSE* NOTICE acme lib rc sys ;
intending to get all the source and not the binaries.  I patched my vac to
ignore atimes (replacing the vac metadata field with the mtime) to increase
metadata block sharing.  As of 2009/0205 (a convenient snapshot to du), this
represents about 140.7 MB of data per dump.  The entire copy takes 550 MB
(240 MB actual storage in Venti).  (With no sharing whatsoever, this would
be approx. 310 GB.)  I would like to re-archive this with the Rabin
fingerprinting vac for comparison.

(In case anybody wants to rush out and recreate the results, it took
roughly 10 to 15 minutes per dump to dispatch all the Tstat requests to
sources.)

Incidentally, a git repository of the crawls, from 2002/1212 to 2009/0205,
is available at http://mirrors.acm.jhu.edu/trees/plan9native/ .  Git gets
the data down to 165M after a gc run, so perhaps it's a better idea than a
venti-based mirror.  I haven't managed to make my version of Uriel's port
(thanks for the start! :) ) of git do the right thing in enough cases yet,
so the git repo may not be updated for a while, but I figured somebody might
want to play with it in the interim.

--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back)

[erik quanstrom]

> (240 MB actual storage in Venti).  (With no sharing whatsoever, this would
> be approx. 310 GB.)  I would like to re-archive this with the Rabin
> fingerprinting vac for comparison.

by "no sharing" do you mean if each file tree were stored in a
seperate fs, or do you mean that the original fs + one copy of
each file each time it has changed is 310GB?  i'm guessing the
former?

- erik

Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back)

[Uriel]

Oh, glad that somebody found my partial git port useful, I might give
it another push some time.

Having a git/hg repo of the plan9 history is something I have been
thinking about for a while, really cool that you got something going
already.

Will you provide a standard git web interface (and a 'native' git
interface for more efficient cloning)?

Peace

uriel

On Wed, Feb 11, 2009 at 2:43 AM, Nathaniel W Filardo <nwf@cs.jhu.edu> wrote:
> On Tue, Feb 10, 2009 at 02:45:43PM -0800, Roman V. Shaposhnik wrote:
>> On Tue, 2009-02-10 at 17:28 -0500, erik quanstrom wrote:
>> > what leads you to beleve that that amount of sharing will be
>> > significant?
>>
>> Just a hunch so far. I don't have hard data to prove anything.
>> On the other hand, I'd be surprised if massive updates (not pulling
>> in a couple of months) didn't benefit from the sharing.
>>
>> Thanks,
>> Roman.
>
> I have mirrored, with vac -f, every sources dump from 2002 to
> yesterday with
>      -e acme/acid/386 -e acme/acid/alpha -e acme/acid/arm \
>      -e acme/acid/mips -e acme/acid/power -e acme/bin/386 \
>      -e acme/bin/alpha -e acme/bin/arm -e acme/bin/mips \
>      -e acme/bin/power -e acme/mail/386 -e acme/mail/alpha \
>      -e acme/mail/arm -e acme/mail/mips -e acme/mail/power \
>      -e sys/man/vol1.ps -e sys/man/vol1.ps.gz -e sys/man/vol1.pdf \
>      LICENSE* NOTICE acme lib rc sys ;
> intending to get all the source and not the binaries.  I patched my vac to
> ignore atimes (replacing the vac metadata field with the mtime) to increase
> metadata block sharing.  As of 2009/0205 (a convenient snapshot to du), this
> represents about 140.7 MB of data per dump.  The entire copy takes 550 MB
> (240 MB actual storage in Venti).  (With no sharing whatsoever, this would
> be approx. 310 GB.)  I would like to re-archive this with the Rabin
> fingerprinting vac for comparison.
>
> (In case anybody wants to rush out and recreate the results, it took
> roughly 10 to 15 minutes per dump to dispatch all the Tstat requests to
> sources.)
>
> Incidentally, a git repository of the crawls, from 2002/1212 to 2009/0205,
> is available at http://mirrors.acm.jhu.edu/trees/plan9native/ .  Git gets
> the data down to 165M after a gc run, so perhaps it's a better idea than a
> venti-based mirror.  I haven't managed to make my version of Uriel's port
> (thanks for the start! :) ) of git do the right thing in enough cases yet,
> so the git repo may not be updated for a while, but I figured somebody might
> want to play with it in the interim.
>
> --nwf;
>

Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back)

[Venkatesh Srinivas]

On Wed, Feb 11, 2009 at 07:07:48PM +0100, Uriel wrote:
>Oh, glad that somebody found my partial git port useful, I might give
>it another push some time.
>
>Having a git/hg repo of the plan9 history is something I have been
>thinking about for a while, really cool that you got something going
>already.
>
>Will you provide a standard git web interface (and a 'native' git
>interface for more efficient cloning)?
>

We'll have a git web interface up pretty soon, within this week.

-- vs

Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back)

[Roman V. Shaposhnik]

On Wed, 2009-02-11 at 13:19 -0500, Venkatesh Srinivas wrote:
> On Wed, Feb 11, 2009 at 07:07:48PM +0100, Uriel wrote:
> >Oh, glad that somebody found my partial git port useful, I might give
> >it another push some time.
> >
> >Having a git/hg repo of the plan9 history is something I have been
> >thinking about for a while, really cool that you got something going
> >already.
> >
> >Will you provide a standard git web interface (and a 'native' git
> >interface for more efficient cloning)?
> >
>
> We'll have a git web interface up pretty soon, within this week.

Since its a more or less r/o Git repo, why not also provide a mirror on
one of these guys:
    http://github.com/
    http://repo.or.cz/
    http://gitorious.org/

Not only would it reduce the stress on your servers, but it'll also
enable some of the source-browsing features that these sites
implement.

I can set things up myself, as long as you give me *some* kind of
access to your Git repo.

Thanks,
Roman.

Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back)

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 647 bytes --]

On Wed, Feb 11, 2009 at 10:35:33AM -0800, Roman V. Shaposhnik wrote:
> Since its a more or less r/o Git repo, why not also provide a mirror on
> one of these guys:
>     http://github.com/
>     http://repo.or.cz/
>     http://gitorious.org/

I don't want to make it even that official yet, in case somebody suggests
changes, e.g. that I've missed files in my crawls (it occurred to me that
I've missed the contents of dist/, if that bothers anybody).  Revising
history like this will break git's history, naturally, and so probably
merits throwing away repos and restarting.  But eventually, it does seem
like a good idea.

--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back)

[Venkatesh Srinivas]

On Wed, Feb 11, 2009 at 07:07:48PM +0100, Uriel wrote:
>Oh, glad that somebody found my partial git port useful, I might give
>it another push some time.
>
>Having a git/hg repo of the plan9 history is something I have been
>thinking about for a while, really cool that you got something going
>already.
>
>Will you provide a standard git web interface (and a 'native' git
>interface for more efficient cloning)?

http://acm.jhu.edu/git/plan9 is a git web interface, git://acm.jhu.edu/git/plan9
is a native git interface.

-- vs

Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back)

[Roman V. Shaposhnik]

On Tue, 2009-02-10 at 20:43 -0500, Nathaniel W Filardo wrote:
> Incidentally, a git repository of the crawls, from 2002/1212 to 2009/0205,
> is available at http://mirrors.acm.jhu.edu/trees/plan9native/ .  Git gets
> the data down to 165M after a gc run, so perhaps it's a better idea than a
> venti-based mirror.

Where did 165M came from? The history itself seems to be only about 58M
or so:
  $ wget http://mirrors.acm.jhu.edu/trees/plan9native/.git/objects/pack/pack-afe021812ab52f698895941f8eb5ad4e3d75020e.pack
  $ ls -l pack-afe021812ab52f698895941f8eb5ad4e3d75020e.pack
  -rw-rw-r--   1 rs76089  staff    61039150 Feb 11 06:40 pack-afe021812ab52f698895941f8eb5ad4e3d75020e.pack

And, after the following simple minded manipulations:
  $ git init
  $ git unpack-objects < pack*
  $ git checkout -b master 68e58814202bccfbd7186962daedd754ae76d7df
  warning: You appear to be on a branch yet to be born.
  warning: Forcing checkout of 68e58814202bccfbd7186962daedd754ae76d7df.
  Checking out files: 100% (14229/14229), done.
  Already on "master"
  $ git repack -ad --window 100 --depth 100
  Counting objects: 39971, done.
  Compressing objects: 100% (39354/39354), done.
  Writing objects: 100% (39971/39971), done.
  Total 39971 (delta 25278), reused 0 (delta 0)

Made it even smaller (you can fine tune it even more, based on
usage requirements):
  $ ls -l .git/objects/pack/*.pack
  -r--r--r--   1 rs76089  staff    57694396 Feb 11 11:03 .git/objects/pack/pack-afe021812ab52f698895941f8eb5ad4e3d75020e.pack

>  I haven't managed to make my version of Uriel's port
> (thanks for the start! :) ) of git do the right thing in enough cases yet,
> so the git repo may not be updated for a while, but I figured somebody might
> want to play with it in the interim.

The coolest things, of course, would be to have a way of running git on
the bell labs end. But doing a replica and repacking everything locally
is not bad at all.

Thanks,
Roman.

Re: [9fans] source browsing via http is back

[sqweek]

On Wed, Feb 11, 2009 at 7:28 AM, erik quanstrom <quanstro@quanstro.net> wrote:
>> > what extra work would that be,
>>
>> that it pulls files in their entirety (IOW: cp /n/sources/... /)
>
> i would be very suprised if a single copy of sources had many shared
> blocks.

 He doesn't want a single copy though, he is hoping to mirror
sourcesdump. That's 2255 almost-copies of sources. Yes, there will be
changes in each snapshot, but how often does something like
/extra/python.iso.bz2 change?
-sqweek

Re: [9fans] source browsing via http is back

[erik quanstrom]

> On Wed, Feb 11, 2009 at 7:28 AM, erik quanstrom <quanstro@quanstro.net> wrote:
> >> > what extra work would that be,
> >>
> >> that it pulls files in their entirety (IOW: cp /n/sources/... /)
> >
> > i would be very suprised if a single copy of sources had many shared
> > blocks.
>
>  He doesn't want a single copy though, he is hoping to mirror
> sourcesdump. That's 2255 almost-copies of sources. Yes, there will be
> changes in each snapshot, but how often does something like
> /extra/python.iso.bz2 change?

exactly.  the point i was trying to make, and evidently
was being too coy about, is that 330 odd gb wouldn't
be as useful a number as the sum of the sizes of all the
new/changed files from all the dump days.  this would
be a useful comparison because this would give a
measure of how much space is saved with venti over
the straightforward algorithm of copying the changed
blocks, as ken's fileserver does.

- erik

Re: [9fans] source browsing via http is back

[Bruce Ellis]

Stop being sensible! There is no room on this list for such behaviour.

brucee

On Thu, Feb 12, 2009 at 11:49 PM, erik quanstrom <quanstro@quanstro.net> wrote:
>> On Wed, Feb 11, 2009 at 7:28 AM, erik quanstrom <quanstro@quanstro.net> wrote:
>> >> > what extra work would that be,
>> >>
>> >> that it pulls files in their entirety (IOW: cp /n/sources/... /)
>> >
>> > i would be very suprised if a single copy of sources had many shared
>> > blocks.
>>
>>  He doesn't want a single copy though, he is hoping to mirror
>> sourcesdump. That's 2255 almost-copies of sources. Yes, there will be
>> changes in each snapshot, but how often does something like
>> /extra/python.iso.bz2 change?
>
> exactly.  the point i was trying to make, and evidently
> was being too coy about, is that 330 odd gb wouldn't
> be as useful a number as the sum of the sizes of all the
> new/changed files from all the dump days.  this would
> be a useful comparison because this would give a
> measure of how much space is saved with venti over
> the straightforward algorithm of copying the changed
> blocks, as ken's fileserver does.
>
> - erik
>
>

Re: [9fans] source browsing via http is back

[Roman V. Shaposhnik]

On Thu, 2009-02-12 at 07:49 -0500, erik quanstrom wrote:
> exactly.  the point i was trying to make, and evidently
> was being too coy about, is that 330 odd gb wouldn't
> be as useful a number as the sum of the sizes of all the
> new/changed files from all the dump days.  this would
> be a useful comparison because this would give a
> measure of how much space is saved with venti over
> the straightforward algorithm of copying the changed
> blocks, as ken's fileserver does.

I'm confused. Since when did kenfs entered this conversation?
I thought we were talking about how sources are managed today
and how replica might make you waste some bandwidth (albeit,
not all that much given how infrequently sources themselves
change).

Thanks,
Roman.

Re: [9fans] source browsing via http is back

[erik quanstrom]

> On Thu, 2009-02-12 at 07:49 -0500, erik quanstrom wrote:
> > exactly.  the point i was trying to make, and evidently
> > was being too coy about, is that 330 odd gb wouldn't
> > be as useful a number as the sum of the sizes of all the
> > new/changed files from all the dump days.  this would
> > be a useful comparison because this would give a
> > measure of how much space is saved with venti over
> > the straightforward algorithm of copying the changed
> > blocks, as ken's fileserver does.
>
> I'm confused. Since when did kenfs entered this conversation?
> I thought we were talking about how sources are managed today
> and how replica might make you waste some bandwidth (albeit,
> not all that much given how infrequently sources themselves
> change).

exactly.  i don't believe that the amount of data replica would
transfer is 330gb.  330gb seems more than replica would transfer.
it's over 1000x the size of the distribution.  i'm pretty sure sources
doesn't see that much churn.

the algorithm replica uses will result in about the same amount
data moved as kenfs would store, as they use similar algorithms.
old unix dump would use a similar amount of space.  mkfs would
use more, as it is file, not block based.

- erik

Re: [9fans] source browsing via http is back

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 2693 bytes --]

On Thu, Feb 12, 2009 at 07:49:58AM -0500, erik quanstrom wrote:
> exactly.  the point i was trying to make, and evidently
> was being too coy about, is that 330 odd gb wouldn't
> be as useful a number as the sum of the sizes of all the
> new/changed files from all the dump days.  this would
> be a useful comparison because this would give a
> measure of how much space is saved with venti over
> the straightforward algorithm of copying the changed
> blocks, as ken's fileserver does.

Unless I misunderstand how replica works, the 330 odd GB number [1] is
useful as the amount of data that would have to be transfered over the wire
to initialize a mirror.  (Since, as I understand it, a replica log of
sourcesdump would have nothing but "add" commands for each $year/$dump/$file
entry, and would therefore necessitate transfering each file separately).

On the other hand, it's entirely possible that I'm missing some feature of
replica, or that some set of wrapper scripts around it would suffice.  If
so, please excuse, and correct, my ignorance.

On the first hand again, given the occasional reports of "replica hosed me"
I'm not terribly keen on trusting it and seem to recall that some of the
fixes have involved hand-editing the replica logs on sources.  This makes me
suspicious that some of the replica logs frozen in sourcesdump would be
incorrect and lead to incorrect data on mirrors if used as part of the
scheme.

With a venti & vac (auth/none vac, naturally, so as to not violate
filesystem permissions) based mirror, there's a single score published daily
that covers the entirety of sourcesdump so far, and a venti/copy -f
sufficies to bring any mirror up to date using at most 550 odd MB if the
initial mirror is empty. [2]

--nwf;

[1] The discrepency between 550 MB and 330 GB increases as time goes on and
as the slice of sources being mirrored goes from "just some source files
that some schmo thought would be nice to mirror" to "all of it".

[2] Further, 9fs access to sources is grand, but it does take me 10 to 15
minutes to pull down a day's worth of "just some source files", even if
nothing has changed and I uses vac -f, due to all the network latency for
Tstat/Rstat requests.  This could be improved in a number of ways, but it
strikes me as simpler to use venti/copy to copy only the incremental deltas.
Some brief experiments, transfering blocks from Baltimore back to a machine
in the same neighborhood as sources, indicate that venti/copy -f takes 15
minutes for the first copy (2002/1212) and that subsequently copying even a
dump with many changes (2008/0901) took only four.  (Git may do even
better.)

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] source browsing via http is back

[andrey mirtchovski]

> On the first hand again, given the occasional reports of "replica hosed me"
> I'm not terribly keen on trusting

given the occasional reports of "software X hosed me" (for any and all
X), i don't think we should be terribly keen on using computers at
all.

Re: [9fans] source browsing via http is back

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

On Thu, Feb 12, 2009 at 09:50:50AM -0700, andrey mirtchovski wrote:
> > On the first hand again, given the occasional reports of "replica hosed me"
> > I'm not terribly keen on trusting
> 
> given the occasional reports of "software X hosed me" (for any and all
> X), i don't think we should be terribly keen on using computers at
> all.

Touche. :)

--nwf; 

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] source browsing via http is back

[erik quanstrom]

> > On the first hand again, given the occasional reports of "replica hosed me"
> > I'm not terribly keen on trusting
>
> given the occasional reports of "software X hosed me" (for any and all
> X), i don't think we should be terribly keen on using computers at
> all.

or, being a programmer, one could endeavor to fix these problems.
i haven't yet found any significant software that worked under all conditions
for all people from the time it was first compiled.

- erik

Re: [9fans] source browsing via http is back

[Bruce Ellis]

bellisimo!

On Fri, Feb 13, 2009 at 3:50 AM, andrey mirtchovski
<mirtchovski@gmail.com> wrote:
>> On the first hand again, given the occasional reports of "replica hosed me"
>> I'm not terribly keen on trusting
>
> given the occasional reports of "software X hosed me" (for any and all
> X), i don't think we should be terribly keen on using computers at
> all.
>
>

Re: [9fans] source browsing via http is back

[erik quanstrom]

> On Thu, Feb 12, 2009 at 07:49:58AM -0500, erik quanstrom wrote:
> > exactly.  the point i was trying to make, and evidently
> > was being too coy about, is that 330 odd gb wouldn't
> > be as useful a number as the sum of the sizes of all the
> > new/changed files from all the dump days.  this would
> > be a useful comparison because this would give a
> > measure of how much space is saved with venti over
> > the straightforward algorithm of copying the changed
> > blocks, as ken's fileserver does.
>
> Unless I misunderstand how replica works, the 330 odd GB number [1] is
> useful as the amount of data that would have to be transfered over the wire
> to initialize a mirror.  (Since, as I understand it, a replica log of
> sourcesdump would have nothing but "add" commands for each $year/$dump/$file
> entry, and would therefore necessitate transfering each file separately).

actually not, see http://www.quanstro.net/plan9/history.pdf
i have done the process of replicating history on 4 different
fileservers.  only the *differences* need to be transfered.

> On the first hand again, given the occasional reports of "replica hosed me"
> I'm not terribly keen on trusting it and seem to recall that some of the
> fixes have involved hand-editing the replica logs on sources.  This makes me
> suspicious that some of the replica logs frozen in sourcesdump would be
> incorrect and lead to incorrect data on mirrors if used as part of the
> scheme.

i've posted a number of fixes for specific failure modes reported.
i've done about 10k replicas with my changes without any failures.
try it.  the only way to fix things is to keep working on them.

- erik

Re: [9fans] source browsing via http is back

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 866 bytes --]

On Tue, Feb 10, 2009 at 04:22:42PM -0500, Nathaniel W Filardo wrote:
> > since 9fs never stopped working, why could you not just export (with
> > httpd) from your machine a -C mount of sources? then the robots hammer
> > you but since you're cached, you don't hammer sources?
> 
> Sorry to rain on your parade as it were, but occasionally 9fs to sources
> does, in fact, stop working.  See my and others' posts to 9fans about
> sources and especially sources' venti being down.

Rereading my own mail, that sounded remarkably ungrateful and I apologize.
I was intending only to suggest that true mirrors may be better than
depending on sources' continuous uptime (and introducing additional services
that break if sources goes down, even for scheduled maintainence).

Thanks again to the labs for all their work for supporting the community.
--nwf; 

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]