* [9fans] source browsing via http is back @ 2009-02-10 18:49 geoff 2009-02-10 19:02 ` Bruce Ellis 2009-02-10 21:10 ` John Barham 0 siblings, 2 replies; 40+ messages in thread From: geoff @ 2009-02-10 18:49 UTC (permalink / raw) To: 9fans I've turned it back on and will watch to see if our web server gets swamped by it. This interface should not be used to mirror the contents of sources. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 18:49 [9fans] source browsing via http is back geoff @ 2009-02-10 19:02 ` Bruce Ellis 2009-02-10 21:10 ` John Barham 1 sibling, 0 replies; 40+ messages in thread From: Bruce Ellis @ 2009-02-10 19:02 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs Thank you geoff. You are one of the good guys. And I've never heard you whine. Now, back to work for brucee and Tiger. brucee On Wed, Feb 11, 2009 at 5:49 AM, <geoff@plan9.bell-labs.com> wrote: > I've turned it back on and will watch to see if our web server gets > swamped by it. This interface should not be used to mirror the > contents of sources. > > ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 18:49 [9fans] source browsing via http is back geoff 2009-02-10 19:02 ` Bruce Ellis @ 2009-02-10 21:10 ` John Barham 2009-02-10 21:15 ` ron minnich 1 sibling, 1 reply; 40+ messages in thread From: John Barham @ 2009-02-10 21:10 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs > I've turned it back on and will watch to see if our web server gets > swamped by it. This interface should not be used to mirror the > contents of sources. What interface should be used to used to mirror sources? 9fs? replica? More specifically, if I wanted to set up my own HTTP mirror of sources, what is the labs approved way to stay in sync? ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 21:10 ` John Barham @ 2009-02-10 21:15 ` ron minnich 2009-02-10 21:22 ` Nathaniel W Filardo 0 siblings, 1 reply; 40+ messages in thread From: ron minnich @ 2009-02-10 21:15 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Tue, Feb 10, 2009 at 1:10 PM, John Barham <jbarham@gmail.com> wrote: >> I've turned it back on and will watch to see if our web server gets >> swamped by it. This interface should not be used to mirror the >> contents of sources. > > What interface should be used to used to mirror sources? 9fs? > replica? More specifically, if I wanted to set up my own HTTP mirror > of sources, what is the labs approved way to stay in sync? > > since 9fs never stopped working, why could you not just export (with httpd) from your machine a -C mount of sources? then the robots hammer you but since you're cached, you don't hammer sources? ron ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 21:15 ` ron minnich @ 2009-02-10 21:22 ` Nathaniel W Filardo 2009-02-10 21:32 ` erik quanstrom 2009-02-10 22:27 ` Nathaniel W Filardo 0 siblings, 2 replies; 40+ messages in thread From: Nathaniel W Filardo @ 2009-02-10 21:22 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 1606 bytes --] On Tue, Feb 10, 2009 at 01:15:11PM -0800, ron minnich wrote: > On Tue, Feb 10, 2009 at 1:10 PM, John Barham <jbarham@gmail.com> wrote: > >> I've turned it back on and will watch to see if our web server gets > >> swamped by it. This interface should not be used to mirror the > >> contents of sources. > > > > What interface should be used to used to mirror sources? 9fs? > > replica? More specifically, if I wanted to set up my own HTTP mirror > > of sources, what is the labs approved way to stay in sync? > > > > > > since 9fs never stopped working, why could you not just export (with > httpd) from your machine a -C mount of sources? then the robots hammer > you but since you're cached, you don't hammer sources? Sorry to rain on your parade as it were, but occasionally 9fs to sources does, in fact, stop working. See my and others' posts to 9fans about sources and especially sources' venti being down. I again propose that sources should be mirrorable via venti (and venti/copy -f); the nightly snapshots would be walked with "auth/none vac" into a publically readable venti (venti/ro proxy) and the scores published (and signed). In order to make this more palatable, it may be worth developing a venti proxy that attempts to pull blocks from closer ventis rather than further ones and to make sources' public venti artificially further away (induce a delay of half a second or something per Tread). I've some tentative writeup on https://wiki.ietfng.org/pub/Plan9/VentiMirror and even some code flying locally but nothing about which to be proud. --nwf; [-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 21:22 ` Nathaniel W Filardo @ 2009-02-10 21:32 ` erik quanstrom 2009-02-10 21:51 ` Roman V. Shaposhnik 2009-02-10 22:08 ` Nathaniel W Filardo 2009-02-10 22:27 ` Nathaniel W Filardo 1 sibling, 2 replies; 40+ messages in thread From: erik quanstrom @ 2009-02-10 21:32 UTC (permalink / raw) To: 9fans > I again propose that sources should be mirrorable via venti (and venti/copy > -f); the nightly snapshots would be walked with "auth/none vac" into a > publically readable venti (venti/ro proxy) and the scores published (and > signed). In order to make this more palatable, it may be worth developing a > venti proxy that attempts to pull blocks from closer ventis rather than > further ones and to make sources' public venti artificially further away > (induce a delay of half a second or something per Tread). I've some > tentative writeup on https://wiki.ietfng.org/pub/Plan9/VentiMirror and even > some code flying locally but nothing about which to be proud. i'm still not following why replica won't work? getting in underneath the fs seems to require some extra justification and it seems to require some very low-level modifications. and yet the file interface provides what i think one would need for such a project. what is wrong with replica that can't be reasonably fixed? - erik ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 21:32 ` erik quanstrom @ 2009-02-10 21:51 ` Roman V. Shaposhnik 2009-02-10 21:55 ` erik quanstrom 2009-02-10 22:13 ` Nathaniel W Filardo 2009-02-10 22:08 ` Nathaniel W Filardo 1 sibling, 2 replies; 40+ messages in thread From: Roman V. Shaposhnik @ 2009-02-10 21:51 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Tue, 2009-02-10 at 16:32 -0500, erik quanstrom wrote: > > I again propose that sources should be mirrorable via venti (and venti/copy > > -f); the nightly snapshots would be walked with "auth/none vac" into a > > publically readable venti (venti/ro proxy) and the scores published (and > > signed). In order to make this more palatable, it may be worth developing a > > venti proxy that attempts to pull blocks from closer ventis rather than > > further ones and to make sources' public venti artificially further away > > (induce a delay of half a second or something per Tread). I've some > > tentative writeup on https://wiki.ietfng.org/pub/Plan9/VentiMirror and even > > some code flying locally but nothing about which to be proud. > > i'm still not following why replica won't work? getting in underneath > the fs seems to require some extra justification and it seems to require > some very low-level modifications. and yet the file interface provides > what i think one would need for such a project. what is wrong with > replica that can't be reasonably fixed? since replica requires some (albeit automatic) periodic work on the server end it means that there's one more thing for bell lab folks to care about and maintain. that said, a brand new Venti proxy or what not will probably be even worse for them :-( Thanks, Roman. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 21:51 ` Roman V. Shaposhnik @ 2009-02-10 21:55 ` erik quanstrom 2009-02-10 22:05 ` Roman V. Shaposhnik 2009-02-10 22:13 ` Nathaniel W Filardo 1 sibling, 1 reply; 40+ messages in thread From: erik quanstrom @ 2009-02-10 21:55 UTC (permalink / raw) To: 9fans > > i'm still not following why replica won't work? getting in underneath > > the fs seems to require some extra justification and it seems to require > > some very low-level modifications. and yet the file interface provides > > what i think one would need for such a project. what is wrong with > > replica that can't be reasonably fixed? > > since replica requires some (albeit automatic) periodic work on the > server end it means that there's one more thing for bell lab folks > to care about and maintain. that said, a brand new Venti proxy or > what not will probably be even worse for them :-( the replica logs are already generated and already required for existing functionality — pull — to work. - erik ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 21:55 ` erik quanstrom @ 2009-02-10 22:05 ` Roman V. Shaposhnik 0 siblings, 0 replies; 40+ messages in thread From: Roman V. Shaposhnik @ 2009-02-10 22:05 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Tue, 2009-02-10 at 16:55 -0500, erik quanstrom wrote: > > > i'm still not following why replica won't work? getting in underneath > > > the fs seems to require some extra justification and it seems to require > > > some very low-level modifications. and yet the file interface provides > > > what i think one would need for such a project. what is wrong with > > > replica that can't be reasonably fixed? > > > > since replica requires some (albeit automatic) periodic work on the > > server end it means that there's one more thing for bell lab folks > > to care about and maintain. that said, a brand new Venti proxy or > > what not will probably be even worse for them :-( > > the replica logs are already generated and already required > for existing functionality — pull — to work. What do you mean "they are already generated"? The script(s) try to generate them. When they fail they need a manual intervention. That was my entire point. Thanks, Roman. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 21:51 ` Roman V. Shaposhnik 2009-02-10 21:55 ` erik quanstrom @ 2009-02-10 22:13 ` Nathaniel W Filardo 2009-02-10 22:17 ` Roman V. Shaposhnik 1 sibling, 1 reply; 40+ messages in thread From: Nathaniel W Filardo @ 2009-02-10 22:13 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 575 bytes --] On Tue, Feb 10, 2009 at 01:51:47PM -0800, Roman V. Shaposhnik wrote: > since replica requires some (albeit automatic) periodic work on the > server end it means that there's one more thing for bell lab folks > to care about and maintain. that said, a brand new Venti proxy or > what not will probably be even worse for them :-( Under the proposed scheme, clients and maybe mirrors, not sources, run the new proxy. Sources runs only a venti and venti/ro, unmodified, to allow clients and mirrors to fetch blocks, and auth/none vac. Is this unreasonable? --nwf; [-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 22:13 ` Nathaniel W Filardo @ 2009-02-10 22:17 ` Roman V. Shaposhnik 0 siblings, 0 replies; 40+ messages in thread From: Roman V. Shaposhnik @ 2009-02-10 22:17 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Tue, 2009-02-10 at 17:13 -0500, Nathaniel W Filardo wrote: > On Tue, Feb 10, 2009 at 01:51:47PM -0800, Roman V. Shaposhnik wrote: > > since replica requires some (albeit automatic) periodic work on the > > server end it means that there's one more thing for bell lab folks > > to care about and maintain. that said, a brand new Venti proxy or > > what not will probably be even worse for them :-( > > Under the proposed scheme, clients and maybe mirrors, not sources, run the > new proxy. Sources runs only a venti and venti/ro, unmodified, to allow > clients and mirrors to fetch blocks, and auth/none vac. Is this unreasonable? It would be completely reasonable (and I wager *less* work for bell labs folks than keeping an eye on replica) if that venti was only hosting blocks coming from sources. As far as I know that's not the case. Thanks, Roman. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 21:32 ` erik quanstrom 2009-02-10 21:51 ` Roman V. Shaposhnik @ 2009-02-10 22:08 ` Nathaniel W Filardo 2009-02-10 22:10 ` erik quanstrom 1 sibling, 1 reply; 40+ messages in thread From: Nathaniel W Filardo @ 2009-02-10 22:08 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 542 bytes --] On Tue, Feb 10, 2009 at 04:32:18PM -0500, erik quanstrom wrote: > i'm still not following why replica won't work? getting in underneath > the fs seems to require some extra justification and it seems to require > some very low-level modifications. and yet the file interface provides > what i think one would need for such a project. what is wrong with > replica that can't be reasonably fixed? I wish to enable mirrors of sourcesdump as well. Replica would do lots of extra work and transfer, vs. venti/copy -f, yes? --nwf; [-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 22:08 ` Nathaniel W Filardo @ 2009-02-10 22:10 ` erik quanstrom 2009-02-10 22:23 ` Roman V. Shaposhnik 0 siblings, 1 reply; 40+ messages in thread From: erik quanstrom @ 2009-02-10 22:10 UTC (permalink / raw) To: 9fans > On Tue, Feb 10, 2009 at 04:32:18PM -0500, erik quanstrom wrote: > > i'm still not following why replica won't work? getting in underneath > > the fs seems to require some extra justification and it seems to require > > some very low-level modifications. and yet the file interface provides > > what i think one would need for such a project. what is wrong with > > replica that can't be reasonably fixed? > > I wish to enable mirrors of sourcesdump as well. Replica would do lots of > extra work and transfer, vs. venti/copy -f, yes? what extra work would that be, and if there is extra work, could you explain why this would be a problem? - erik ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 22:10 ` erik quanstrom @ 2009-02-10 22:23 ` Roman V. Shaposhnik 2009-02-10 22:28 ` erik quanstrom 0 siblings, 1 reply; 40+ messages in thread From: Roman V. Shaposhnik @ 2009-02-10 22:23 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Tue, 2009-02-10 at 17:10 -0500, erik quanstrom wrote: > > On Tue, Feb 10, 2009 at 04:32:18PM -0500, erik quanstrom wrote: > > > i'm still not following why replica won't work? getting in underneath > > > the fs seems to require some extra justification and it seems to require > > > some very low-level modifications. and yet the file interface provides > > > what i think one would need for such a project. what is wrong with > > > replica that can't be reasonably fixed? > > > > I wish to enable mirrors of sourcesdump as well. Replica would do lots of > > extra work and transfer, vs. venti/copy -f, yes? > > what extra work would that be, and if there is extra work, could you explain > why this would be a problem? Correct me if I'm wrong, but my understanding of how pull works is that it pulls files in their entirety (IOW: cp /n/sources/... /) which means that shared blocks get copied as many times as there are files sharing them. Thanks, Roman. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 22:23 ` Roman V. Shaposhnik @ 2009-02-10 22:28 ` erik quanstrom 2009-02-10 22:45 ` Roman V. Shaposhnik 2009-02-12 5:57 ` [9fans] source browsing via http is back sqweek 0 siblings, 2 replies; 40+ messages in thread From: erik quanstrom @ 2009-02-10 22:28 UTC (permalink / raw) To: 9fans > > what extra work would that be, and if there is extra work, could you explain > > why this would be a problem? > > Correct me if I'm wrong, but my understanding of how pull works is > that it pulls files in their entirety (IOW: cp /n/sources/... /) > which means that shared blocks get copied as many times as there > are files sharing them. i would be very suprised if a single copy of sources had many shared blocks. furthermore, as is made quite plain by upas, a change early in a file almost always changes every subsequent block in the file. there are some corner cases for block-sized or same sized adds/deletes. but nontheless, the vast majority of changes will change a good deal of of the blocks comprising the file. what leads you to beleve that that amount of sharing will be significant? - erik ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 22:28 ` erik quanstrom @ 2009-02-10 22:45 ` Roman V. Shaposhnik 2009-02-11 0:22 ` Bruce Ellis 2009-02-11 1:43 ` [9fans] Plan 9 source history (was: Re: source browsing via http is back) Nathaniel W Filardo 2009-02-12 5:57 ` [9fans] source browsing via http is back sqweek 1 sibling, 2 replies; 40+ messages in thread From: Roman V. Shaposhnik @ 2009-02-10 22:45 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Tue, 2009-02-10 at 17:28 -0500, erik quanstrom wrote: > what leads you to beleve that that amount of sharing will be > significant? Just a hunch so far. I don't have hard data to prove anything. On the other hand, I'd be surprised if massive updates (not pulling in a couple of months) didn't benefit from the sharing. Thanks, Roman. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 22:45 ` Roman V. Shaposhnik @ 2009-02-11 0:22 ` Bruce Ellis 2009-02-11 0:28 ` Roman V. Shaposhnik 2009-02-11 0:32 ` Akshat Kumar 2009-02-11 1:43 ` [9fans] Plan 9 source history (was: Re: source browsing via http is back) Nathaniel W Filardo 1 sibling, 2 replies; 40+ messages in thread From: Bruce Ellis @ 2009-02-11 0:22 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs get over it. it works fine on my found machine. and my $300 eeePC. what exactly doesn't work? brucee On Wed, Feb 11, 2009 at 9:45 AM, Roman V. Shaposhnik <rvs@sun.com> wrote: > On Tue, 2009-02-10 at 17:28 -0500, erik quanstrom wrote: >> what leads you to beleve that that amount of sharing will be >> significant? > > Just a hunch so far. I don't have hard data to prove anything. > On the other hand, I'd be surprised if massive updates (not pulling > in a couple of months) didn't benefit from the sharing. > > Thanks, > Roman. > > > ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-11 0:22 ` Bruce Ellis @ 2009-02-11 0:28 ` Roman V. Shaposhnik 2009-02-11 6:06 ` Bruce Ellis 2009-02-11 0:32 ` Akshat Kumar 1 sibling, 1 reply; 40+ messages in thread From: Roman V. Shaposhnik @ 2009-02-11 0:28 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, 2009-02-11 at 11:22 +1100, Bruce Ellis wrote: > get over it. it works fine on my found machine. and my $300 eeePC. > what exactly doesn't work? no argument there. my automechanic still uses MS DOS 5.0. works great for him. i tried to tell him about plan9, but strangely enough he also told me to get over it. Thanks, Roman. p.s. oh, and he's charges the least for the most reasonable service in the area. think coincidence? i think not! ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-11 0:28 ` Roman V. Shaposhnik @ 2009-02-11 6:06 ` Bruce Ellis 0 siblings, 0 replies; 40+ messages in thread From: Bruce Ellis @ 2009-02-11 6:06 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs i have no problem with pragmatism. brucee On Wed, Feb 11, 2009 at 11:28 AM, Roman V. Shaposhnik <rvs@sun.com> wrote: > On Wed, 2009-02-11 at 11:22 +1100, Bruce Ellis wrote: >> get over it. it works fine on my found machine. and my $300 eeePC. >> what exactly doesn't work? > > no argument there. my automechanic still uses MS DOS 5.0. works > great for him. i tried to tell him about plan9, but strangely > enough he also told me to get over it. > > Thanks, > Roman. > > p.s. oh, and he's charges the least for the most reasonable > service in the area. think coincidence? i think not! > > > ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-11 0:22 ` Bruce Ellis 2009-02-11 0:28 ` Roman V. Shaposhnik @ 2009-02-11 0:32 ` Akshat Kumar 1 sibling, 0 replies; 40+ messages in thread From: Akshat Kumar @ 2009-02-11 0:32 UTC (permalink / raw) To: 9fans > it works fine on my found machine. and my $300 eeePC. 9vx (built on another computer, and off of an external HDD) works just fine in eeePC. But what are you using for a Plan 9 environment on that thing? in case your milkshake's better than mine ak ^ permalink raw reply [flat|nested] 40+ messages in thread
* [9fans] Plan 9 source history (was: Re: source browsing via http is back) 2009-02-10 22:45 ` Roman V. Shaposhnik 2009-02-11 0:22 ` Bruce Ellis @ 2009-02-11 1:43 ` Nathaniel W Filardo 2009-02-11 3:40 ` erik quanstrom ` (2 more replies) 1 sibling, 3 replies; 40+ messages in thread From: Nathaniel W Filardo @ 2009-02-11 1:43 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 2091 bytes --] On Tue, Feb 10, 2009 at 02:45:43PM -0800, Roman V. Shaposhnik wrote: > On Tue, 2009-02-10 at 17:28 -0500, erik quanstrom wrote: > > what leads you to beleve that that amount of sharing will be > > significant? > > Just a hunch so far. I don't have hard data to prove anything. > On the other hand, I'd be surprised if massive updates (not pulling > in a couple of months) didn't benefit from the sharing. > > Thanks, > Roman. I have mirrored, with vac -f, every sources dump from 2002 to yesterday with -e acme/acid/386 -e acme/acid/alpha -e acme/acid/arm \ -e acme/acid/mips -e acme/acid/power -e acme/bin/386 \ -e acme/bin/alpha -e acme/bin/arm -e acme/bin/mips \ -e acme/bin/power -e acme/mail/386 -e acme/mail/alpha \ -e acme/mail/arm -e acme/mail/mips -e acme/mail/power \ -e sys/man/vol1.ps -e sys/man/vol1.ps.gz -e sys/man/vol1.pdf \ LICENSE* NOTICE acme lib rc sys ; intending to get all the source and not the binaries. I patched my vac to ignore atimes (replacing the vac metadata field with the mtime) to increase metadata block sharing. As of 2009/0205 (a convenient snapshot to du), this represents about 140.7 MB of data per dump. The entire copy takes 550 MB (240 MB actual storage in Venti). (With no sharing whatsoever, this would be approx. 310 GB.) I would like to re-archive this with the Rabin fingerprinting vac for comparison. (In case anybody wants to rush out and recreate the results, it took roughly 10 to 15 minutes per dump to dispatch all the Tstat requests to sources.) Incidentally, a git repository of the crawls, from 2002/1212 to 2009/0205, is available at http://mirrors.acm.jhu.edu/trees/plan9native/ . Git gets the data down to 165M after a gc run, so perhaps it's a better idea than a venti-based mirror. I haven't managed to make my version of Uriel's port (thanks for the start! :) ) of git do the right thing in enough cases yet, so the git repo may not be updated for a while, but I figured somebody might want to play with it in the interim. --nwf; [-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back) 2009-02-11 1:43 ` [9fans] Plan 9 source history (was: Re: source browsing via http is back) Nathaniel W Filardo @ 2009-02-11 3:40 ` erik quanstrom 2009-02-11 18:07 ` Uriel 2009-02-11 19:06 ` Roman V. Shaposhnik 2 siblings, 0 replies; 40+ messages in thread From: erik quanstrom @ 2009-02-11 3:40 UTC (permalink / raw) To: 9fans > (240 MB actual storage in Venti). (With no sharing whatsoever, this would > be approx. 310 GB.) I would like to re-archive this with the Rabin > fingerprinting vac for comparison. by "no sharing" do you mean if each file tree were stored in a seperate fs, or do you mean that the original fs + one copy of each file each time it has changed is 310GB? i'm guessing the former? - erik ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back) 2009-02-11 1:43 ` [9fans] Plan 9 source history (was: Re: source browsing via http is back) Nathaniel W Filardo 2009-02-11 3:40 ` erik quanstrom @ 2009-02-11 18:07 ` Uriel 2009-02-11 18:19 ` Venkatesh Srinivas 2009-02-12 15:10 ` Venkatesh Srinivas 2009-02-11 19:06 ` Roman V. Shaposhnik 2 siblings, 2 replies; 40+ messages in thread From: Uriel @ 2009-02-11 18:07 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs Oh, glad that somebody found my partial git port useful, I might give it another push some time. Having a git/hg repo of the plan9 history is something I have been thinking about for a while, really cool that you got something going already. Will you provide a standard git web interface (and a 'native' git interface for more efficient cloning)? Peace uriel On Wed, Feb 11, 2009 at 2:43 AM, Nathaniel W Filardo <nwf@cs.jhu.edu> wrote: > On Tue, Feb 10, 2009 at 02:45:43PM -0800, Roman V. Shaposhnik wrote: >> On Tue, 2009-02-10 at 17:28 -0500, erik quanstrom wrote: >> > what leads you to beleve that that amount of sharing will be >> > significant? >> >> Just a hunch so far. I don't have hard data to prove anything. >> On the other hand, I'd be surprised if massive updates (not pulling >> in a couple of months) didn't benefit from the sharing. >> >> Thanks, >> Roman. > > I have mirrored, with vac -f, every sources dump from 2002 to > yesterday with > -e acme/acid/386 -e acme/acid/alpha -e acme/acid/arm \ > -e acme/acid/mips -e acme/acid/power -e acme/bin/386 \ > -e acme/bin/alpha -e acme/bin/arm -e acme/bin/mips \ > -e acme/bin/power -e acme/mail/386 -e acme/mail/alpha \ > -e acme/mail/arm -e acme/mail/mips -e acme/mail/power \ > -e sys/man/vol1.ps -e sys/man/vol1.ps.gz -e sys/man/vol1.pdf \ > LICENSE* NOTICE acme lib rc sys ; > intending to get all the source and not the binaries. I patched my vac to > ignore atimes (replacing the vac metadata field with the mtime) to increase > metadata block sharing. As of 2009/0205 (a convenient snapshot to du), this > represents about 140.7 MB of data per dump. The entire copy takes 550 MB > (240 MB actual storage in Venti). (With no sharing whatsoever, this would > be approx. 310 GB.) I would like to re-archive this with the Rabin > fingerprinting vac for comparison. > > (In case anybody wants to rush out and recreate the results, it took > roughly 10 to 15 minutes per dump to dispatch all the Tstat requests to > sources.) > > Incidentally, a git repository of the crawls, from 2002/1212 to 2009/0205, > is available at http://mirrors.acm.jhu.edu/trees/plan9native/ . Git gets > the data down to 165M after a gc run, so perhaps it's a better idea than a > venti-based mirror. I haven't managed to make my version of Uriel's port > (thanks for the start! :) ) of git do the right thing in enough cases yet, > so the git repo may not be updated for a while, but I figured somebody might > want to play with it in the interim. > > --nwf; > ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back) 2009-02-11 18:07 ` Uriel @ 2009-02-11 18:19 ` Venkatesh Srinivas 2009-02-11 18:35 ` Roman V. Shaposhnik 2009-02-12 15:10 ` Venkatesh Srinivas 1 sibling, 1 reply; 40+ messages in thread From: Venkatesh Srinivas @ 2009-02-11 18:19 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, Feb 11, 2009 at 07:07:48PM +0100, Uriel wrote: >Oh, glad that somebody found my partial git port useful, I might give >it another push some time. > >Having a git/hg repo of the plan9 history is something I have been >thinking about for a while, really cool that you got something going >already. > >Will you provide a standard git web interface (and a 'native' git >interface for more efficient cloning)? > We'll have a git web interface up pretty soon, within this week. -- vs ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back) 2009-02-11 18:19 ` Venkatesh Srinivas @ 2009-02-11 18:35 ` Roman V. Shaposhnik 2009-02-11 18:46 ` Nathaniel W Filardo 0 siblings, 1 reply; 40+ messages in thread From: Roman V. Shaposhnik @ 2009-02-11 18:35 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, 2009-02-11 at 13:19 -0500, Venkatesh Srinivas wrote: > On Wed, Feb 11, 2009 at 07:07:48PM +0100, Uriel wrote: > >Oh, glad that somebody found my partial git port useful, I might give > >it another push some time. > > > >Having a git/hg repo of the plan9 history is something I have been > >thinking about for a while, really cool that you got something going > >already. > > > >Will you provide a standard git web interface (and a 'native' git > >interface for more efficient cloning)? > > > > We'll have a git web interface up pretty soon, within this week. Since its a more or less r/o Git repo, why not also provide a mirror on one of these guys: http://github.com/ http://repo.or.cz/ http://gitorious.org/ Not only would it reduce the stress on your servers, but it'll also enable some of the source-browsing features that these sites implement. I can set things up myself, as long as you give me *some* kind of access to your Git repo. Thanks, Roman. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back) 2009-02-11 18:35 ` Roman V. Shaposhnik @ 2009-02-11 18:46 ` Nathaniel W Filardo 0 siblings, 0 replies; 40+ messages in thread From: Nathaniel W Filardo @ 2009-02-11 18:46 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 647 bytes --] On Wed, Feb 11, 2009 at 10:35:33AM -0800, Roman V. Shaposhnik wrote: > Since its a more or less r/o Git repo, why not also provide a mirror on > one of these guys: > http://github.com/ > http://repo.or.cz/ > http://gitorious.org/ I don't want to make it even that official yet, in case somebody suggests changes, e.g. that I've missed files in my crawls (it occurred to me that I've missed the contents of dist/, if that bothers anybody). Revising history like this will break git's history, naturally, and so probably merits throwing away repos and restarting. But eventually, it does seem like a good idea. --nwf; [-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back) 2009-02-11 18:07 ` Uriel 2009-02-11 18:19 ` Venkatesh Srinivas @ 2009-02-12 15:10 ` Venkatesh Srinivas 1 sibling, 0 replies; 40+ messages in thread From: Venkatesh Srinivas @ 2009-02-12 15:10 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, Feb 11, 2009 at 07:07:48PM +0100, Uriel wrote: >Oh, glad that somebody found my partial git port useful, I might give >it another push some time. > >Having a git/hg repo of the plan9 history is something I have been >thinking about for a while, really cool that you got something going >already. > >Will you provide a standard git web interface (and a 'native' git >interface for more efficient cloning)? http://acm.jhu.edu/git/plan9 is a git web interface, git://acm.jhu.edu/git/plan9 is a native git interface. -- vs ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] Plan 9 source history (was: Re: source browsing via http is back) 2009-02-11 1:43 ` [9fans] Plan 9 source history (was: Re: source browsing via http is back) Nathaniel W Filardo 2009-02-11 3:40 ` erik quanstrom 2009-02-11 18:07 ` Uriel @ 2009-02-11 19:06 ` Roman V. Shaposhnik 2 siblings, 0 replies; 40+ messages in thread From: Roman V. Shaposhnik @ 2009-02-11 19:06 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Tue, 2009-02-10 at 20:43 -0500, Nathaniel W Filardo wrote: > Incidentally, a git repository of the crawls, from 2002/1212 to 2009/0205, > is available at http://mirrors.acm.jhu.edu/trees/plan9native/ . Git gets > the data down to 165M after a gc run, so perhaps it's a better idea than a > venti-based mirror. Where did 165M came from? The history itself seems to be only about 58M or so: $ wget http://mirrors.acm.jhu.edu/trees/plan9native/.git/objects/pack/pack-afe021812ab52f698895941f8eb5ad4e3d75020e.pack $ ls -l pack-afe021812ab52f698895941f8eb5ad4e3d75020e.pack -rw-rw-r-- 1 rs76089 staff 61039150 Feb 11 06:40 pack-afe021812ab52f698895941f8eb5ad4e3d75020e.pack And, after the following simple minded manipulations: $ git init $ git unpack-objects < pack* $ git checkout -b master 68e58814202bccfbd7186962daedd754ae76d7df warning: You appear to be on a branch yet to be born. warning: Forcing checkout of 68e58814202bccfbd7186962daedd754ae76d7df. Checking out files: 100% (14229/14229), done. Already on "master" $ git repack -ad --window 100 --depth 100 Counting objects: 39971, done. Compressing objects: 100% (39354/39354), done. Writing objects: 100% (39971/39971), done. Total 39971 (delta 25278), reused 0 (delta 0) Made it even smaller (you can fine tune it even more, based on usage requirements): $ ls -l .git/objects/pack/*.pack -r--r--r-- 1 rs76089 staff 57694396 Feb 11 11:03 .git/objects/pack/pack-afe021812ab52f698895941f8eb5ad4e3d75020e.pack > I haven't managed to make my version of Uriel's port > (thanks for the start! :) ) of git do the right thing in enough cases yet, > so the git repo may not be updated for a while, but I figured somebody might > want to play with it in the interim. The coolest things, of course, would be to have a way of running git on the bell labs end. But doing a replica and repacking everything locally is not bad at all. Thanks, Roman. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 22:28 ` erik quanstrom 2009-02-10 22:45 ` Roman V. Shaposhnik @ 2009-02-12 5:57 ` sqweek 2009-02-12 12:49 ` erik quanstrom 1 sibling, 1 reply; 40+ messages in thread From: sqweek @ 2009-02-12 5:57 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Wed, Feb 11, 2009 at 7:28 AM, erik quanstrom <quanstro@quanstro.net> wrote: >> > what extra work would that be, >> >> that it pulls files in their entirety (IOW: cp /n/sources/... /) > > i would be very suprised if a single copy of sources had many shared > blocks. He doesn't want a single copy though, he is hoping to mirror sourcesdump. That's 2255 almost-copies of sources. Yes, there will be changes in each snapshot, but how often does something like /extra/python.iso.bz2 change? -sqweek ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-12 5:57 ` [9fans] source browsing via http is back sqweek @ 2009-02-12 12:49 ` erik quanstrom 2009-02-12 13:10 ` Bruce Ellis ` (2 more replies) 0 siblings, 3 replies; 40+ messages in thread From: erik quanstrom @ 2009-02-12 12:49 UTC (permalink / raw) To: 9fans > On Wed, Feb 11, 2009 at 7:28 AM, erik quanstrom <quanstro@quanstro.net> wrote: > >> > what extra work would that be, > >> > >> that it pulls files in their entirety (IOW: cp /n/sources/... /) > > > > i would be very suprised if a single copy of sources had many shared > > blocks. > > He doesn't want a single copy though, he is hoping to mirror > sourcesdump. That's 2255 almost-copies of sources. Yes, there will be > changes in each snapshot, but how often does something like > /extra/python.iso.bz2 change? exactly. the point i was trying to make, and evidently was being too coy about, is that 330 odd gb wouldn't be as useful a number as the sum of the sizes of all the new/changed files from all the dump days. this would be a useful comparison because this would give a measure of how much space is saved with venti over the straightforward algorithm of copying the changed blocks, as ken's fileserver does. - erik ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-12 12:49 ` erik quanstrom @ 2009-02-12 13:10 ` Bruce Ellis 2009-02-12 16:19 ` Roman V. Shaposhnik 2009-02-12 16:42 ` Nathaniel W Filardo 2 siblings, 0 replies; 40+ messages in thread From: Bruce Ellis @ 2009-02-12 13:10 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs Stop being sensible! There is no room on this list for such behaviour. brucee On Thu, Feb 12, 2009 at 11:49 PM, erik quanstrom <quanstro@quanstro.net> wrote: >> On Wed, Feb 11, 2009 at 7:28 AM, erik quanstrom <quanstro@quanstro.net> wrote: >> >> > what extra work would that be, >> >> >> >> that it pulls files in their entirety (IOW: cp /n/sources/... /) >> > >> > i would be very suprised if a single copy of sources had many shared >> > blocks. >> >> He doesn't want a single copy though, he is hoping to mirror >> sourcesdump. That's 2255 almost-copies of sources. Yes, there will be >> changes in each snapshot, but how often does something like >> /extra/python.iso.bz2 change? > > exactly. the point i was trying to make, and evidently > was being too coy about, is that 330 odd gb wouldn't > be as useful a number as the sum of the sizes of all the > new/changed files from all the dump days. this would > be a useful comparison because this would give a > measure of how much space is saved with venti over > the straightforward algorithm of copying the changed > blocks, as ken's fileserver does. > > - erik > > ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-12 12:49 ` erik quanstrom 2009-02-12 13:10 ` Bruce Ellis @ 2009-02-12 16:19 ` Roman V. Shaposhnik 2009-02-12 16:28 ` erik quanstrom 2009-02-12 16:42 ` Nathaniel W Filardo 2 siblings, 1 reply; 40+ messages in thread From: Roman V. Shaposhnik @ 2009-02-12 16:19 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On Thu, 2009-02-12 at 07:49 -0500, erik quanstrom wrote: > exactly. the point i was trying to make, and evidently > was being too coy about, is that 330 odd gb wouldn't > be as useful a number as the sum of the sizes of all the > new/changed files from all the dump days. this would > be a useful comparison because this would give a > measure of how much space is saved with venti over > the straightforward algorithm of copying the changed > blocks, as ken's fileserver does. I'm confused. Since when did kenfs entered this conversation? I thought we were talking about how sources are managed today and how replica might make you waste some bandwidth (albeit, not all that much given how infrequently sources themselves change). Thanks, Roman. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-12 16:19 ` Roman V. Shaposhnik @ 2009-02-12 16:28 ` erik quanstrom 0 siblings, 0 replies; 40+ messages in thread From: erik quanstrom @ 2009-02-12 16:28 UTC (permalink / raw) To: 9fans > On Thu, 2009-02-12 at 07:49 -0500, erik quanstrom wrote: > > exactly. the point i was trying to make, and evidently > > was being too coy about, is that 330 odd gb wouldn't > > be as useful a number as the sum of the sizes of all the > > new/changed files from all the dump days. this would > > be a useful comparison because this would give a > > measure of how much space is saved with venti over > > the straightforward algorithm of copying the changed > > blocks, as ken's fileserver does. > > I'm confused. Since when did kenfs entered this conversation? > I thought we were talking about how sources are managed today > and how replica might make you waste some bandwidth (albeit, > not all that much given how infrequently sources themselves > change). exactly. i don't believe that the amount of data replica would transfer is 330gb. 330gb seems more than replica would transfer. it's over 1000x the size of the distribution. i'm pretty sure sources doesn't see that much churn. the algorithm replica uses will result in about the same amount data moved as kenfs would store, as they use similar algorithms. old unix dump would use a similar amount of space. mkfs would use more, as it is file, not block based. - erik ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-12 12:49 ` erik quanstrom 2009-02-12 13:10 ` Bruce Ellis 2009-02-12 16:19 ` Roman V. Shaposhnik @ 2009-02-12 16:42 ` Nathaniel W Filardo 2009-02-12 16:50 ` andrey mirtchovski 2009-02-12 16:52 ` erik quanstrom 2 siblings, 2 replies; 40+ messages in thread From: Nathaniel W Filardo @ 2009-02-12 16:42 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 2693 bytes --] On Thu, Feb 12, 2009 at 07:49:58AM -0500, erik quanstrom wrote: > exactly. the point i was trying to make, and evidently > was being too coy about, is that 330 odd gb wouldn't > be as useful a number as the sum of the sizes of all the > new/changed files from all the dump days. this would > be a useful comparison because this would give a > measure of how much space is saved with venti over > the straightforward algorithm of copying the changed > blocks, as ken's fileserver does. Unless I misunderstand how replica works, the 330 odd GB number [1] is useful as the amount of data that would have to be transfered over the wire to initialize a mirror. (Since, as I understand it, a replica log of sourcesdump would have nothing but "add" commands for each $year/$dump/$file entry, and would therefore necessitate transfering each file separately). On the other hand, it's entirely possible that I'm missing some feature of replica, or that some set of wrapper scripts around it would suffice. If so, please excuse, and correct, my ignorance. On the first hand again, given the occasional reports of "replica hosed me" I'm not terribly keen on trusting it and seem to recall that some of the fixes have involved hand-editing the replica logs on sources. This makes me suspicious that some of the replica logs frozen in sourcesdump would be incorrect and lead to incorrect data on mirrors if used as part of the scheme. With a venti & vac (auth/none vac, naturally, so as to not violate filesystem permissions) based mirror, there's a single score published daily that covers the entirety of sourcesdump so far, and a venti/copy -f sufficies to bring any mirror up to date using at most 550 odd MB if the initial mirror is empty. [2] --nwf; [1] The discrepency between 550 MB and 330 GB increases as time goes on and as the slice of sources being mirrored goes from "just some source files that some schmo thought would be nice to mirror" to "all of it". [2] Further, 9fs access to sources is grand, but it does take me 10 to 15 minutes to pull down a day's worth of "just some source files", even if nothing has changed and I uses vac -f, due to all the network latency for Tstat/Rstat requests. This could be improved in a number of ways, but it strikes me as simpler to use venti/copy to copy only the incremental deltas. Some brief experiments, transfering blocks from Baltimore back to a machine in the same neighborhood as sources, indicate that venti/copy -f takes 15 minutes for the first copy (2002/1212) and that subsequently copying even a dump with many changes (2008/0901) took only four. (Git may do even better.) [-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-12 16:42 ` Nathaniel W Filardo @ 2009-02-12 16:50 ` andrey mirtchovski 2009-02-12 16:56 ` Nathaniel W Filardo ` (2 more replies) 2009-02-12 16:52 ` erik quanstrom 1 sibling, 3 replies; 40+ messages in thread From: andrey mirtchovski @ 2009-02-12 16:50 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs > On the first hand again, given the occasional reports of "replica hosed me" > I'm not terribly keen on trusting given the occasional reports of "software X hosed me" (for any and all X), i don't think we should be terribly keen on using computers at all. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-12 16:50 ` andrey mirtchovski @ 2009-02-12 16:56 ` Nathaniel W Filardo 2009-02-12 16:58 ` erik quanstrom 2009-02-12 17:20 ` Bruce Ellis 2 siblings, 0 replies; 40+ messages in thread From: Nathaniel W Filardo @ 2009-02-12 16:56 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 370 bytes --] On Thu, Feb 12, 2009 at 09:50:50AM -0700, andrey mirtchovski wrote: > > On the first hand again, given the occasional reports of "replica hosed me" > > I'm not terribly keen on trusting > > given the occasional reports of "software X hosed me" (for any and all > X), i don't think we should be terribly keen on using computers at > all. Touche. :) --nwf; [-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-12 16:50 ` andrey mirtchovski 2009-02-12 16:56 ` Nathaniel W Filardo @ 2009-02-12 16:58 ` erik quanstrom 2009-02-12 17:20 ` Bruce Ellis 2 siblings, 0 replies; 40+ messages in thread From: erik quanstrom @ 2009-02-12 16:58 UTC (permalink / raw) To: 9fans > > On the first hand again, given the occasional reports of "replica hosed me" > > I'm not terribly keen on trusting > > given the occasional reports of "software X hosed me" (for any and all > X), i don't think we should be terribly keen on using computers at > all. or, being a programmer, one could endeavor to fix these problems. i haven't yet found any significant software that worked under all conditions for all people from the time it was first compiled. - erik ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-12 16:50 ` andrey mirtchovski 2009-02-12 16:56 ` Nathaniel W Filardo 2009-02-12 16:58 ` erik quanstrom @ 2009-02-12 17:20 ` Bruce Ellis 2 siblings, 0 replies; 40+ messages in thread From: Bruce Ellis @ 2009-02-12 17:20 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs bellisimo! On Fri, Feb 13, 2009 at 3:50 AM, andrey mirtchovski <mirtchovski@gmail.com> wrote: >> On the first hand again, given the occasional reports of "replica hosed me" >> I'm not terribly keen on trusting > > given the occasional reports of "software X hosed me" (for any and all > X), i don't think we should be terribly keen on using computers at > all. > > ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-12 16:42 ` Nathaniel W Filardo 2009-02-12 16:50 ` andrey mirtchovski @ 2009-02-12 16:52 ` erik quanstrom 1 sibling, 0 replies; 40+ messages in thread From: erik quanstrom @ 2009-02-12 16:52 UTC (permalink / raw) To: 9fans > On Thu, Feb 12, 2009 at 07:49:58AM -0500, erik quanstrom wrote: > > exactly. the point i was trying to make, and evidently > > was being too coy about, is that 330 odd gb wouldn't > > be as useful a number as the sum of the sizes of all the > > new/changed files from all the dump days. this would > > be a useful comparison because this would give a > > measure of how much space is saved with venti over > > the straightforward algorithm of copying the changed > > blocks, as ken's fileserver does. > > Unless I misunderstand how replica works, the 330 odd GB number [1] is > useful as the amount of data that would have to be transfered over the wire > to initialize a mirror. (Since, as I understand it, a replica log of > sourcesdump would have nothing but "add" commands for each $year/$dump/$file > entry, and would therefore necessitate transfering each file separately). actually not, see http://www.quanstro.net/plan9/history.pdf i have done the process of replicating history on 4 different fileservers. only the *differences* need to be transfered. > On the first hand again, given the occasional reports of "replica hosed me" > I'm not terribly keen on trusting it and seem to recall that some of the > fixes have involved hand-editing the replica logs on sources. This makes me > suspicious that some of the replica logs frozen in sourcesdump would be > incorrect and lead to incorrect data on mirrors if used as part of the > scheme. i've posted a number of fixes for specific failure modes reported. i've done about 10k replicas with my changes without any failures. try it. the only way to fix things is to keep working on them. - erik ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [9fans] source browsing via http is back 2009-02-10 21:22 ` Nathaniel W Filardo 2009-02-10 21:32 ` erik quanstrom @ 2009-02-10 22:27 ` Nathaniel W Filardo 1 sibling, 0 replies; 40+ messages in thread From: Nathaniel W Filardo @ 2009-02-10 22:27 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 866 bytes --] On Tue, Feb 10, 2009 at 04:22:42PM -0500, Nathaniel W Filardo wrote: > > since 9fs never stopped working, why could you not just export (with > > httpd) from your machine a -C mount of sources? then the robots hammer > > you but since you're cached, you don't hammer sources? > > Sorry to rain on your parade as it were, but occasionally 9fs to sources > does, in fact, stop working. See my and others' posts to 9fans about > sources and especially sources' venti being down. Rereading my own mail, that sounded remarkably ungrateful and I apologize. I was intending only to suggest that true mirrors may be better than depending on sources' continuous uptime (and introducing additional services that break if sources goes down, even for scheduled maintainence). Thanks again to the labs for all their work for supporting the community. --nwf; [-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
end of thread, other threads:[~2009-02-12 17:20 UTC | newest] Thread overview: 40+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2009-02-10 18:49 [9fans] source browsing via http is back geoff 2009-02-10 19:02 ` Bruce Ellis 2009-02-10 21:10 ` John Barham 2009-02-10 21:15 ` ron minnich 2009-02-10 21:22 ` Nathaniel W Filardo 2009-02-10 21:32 ` erik quanstrom 2009-02-10 21:51 ` Roman V. Shaposhnik 2009-02-10 21:55 ` erik quanstrom 2009-02-10 22:05 ` Roman V. Shaposhnik 2009-02-10 22:13 ` Nathaniel W Filardo 2009-02-10 22:17 ` Roman V. Shaposhnik 2009-02-10 22:08 ` Nathaniel W Filardo 2009-02-10 22:10 ` erik quanstrom 2009-02-10 22:23 ` Roman V. Shaposhnik 2009-02-10 22:28 ` erik quanstrom 2009-02-10 22:45 ` Roman V. Shaposhnik 2009-02-11 0:22 ` Bruce Ellis 2009-02-11 0:28 ` Roman V. Shaposhnik 2009-02-11 6:06 ` Bruce Ellis 2009-02-11 0:32 ` Akshat Kumar 2009-02-11 1:43 ` [9fans] Plan 9 source history (was: Re: source browsing via http is back) Nathaniel W Filardo 2009-02-11 3:40 ` erik quanstrom 2009-02-11 18:07 ` Uriel 2009-02-11 18:19 ` Venkatesh Srinivas 2009-02-11 18:35 ` Roman V. Shaposhnik 2009-02-11 18:46 ` Nathaniel W Filardo 2009-02-12 15:10 ` Venkatesh Srinivas 2009-02-11 19:06 ` Roman V. Shaposhnik 2009-02-12 5:57 ` [9fans] source browsing via http is back sqweek 2009-02-12 12:49 ` erik quanstrom 2009-02-12 13:10 ` Bruce Ellis 2009-02-12 16:19 ` Roman V. Shaposhnik 2009-02-12 16:28 ` erik quanstrom 2009-02-12 16:42 ` Nathaniel W Filardo 2009-02-12 16:50 ` andrey mirtchovski 2009-02-12 16:56 ` Nathaniel W Filardo 2009-02-12 16:58 ` erik quanstrom 2009-02-12 17:20 ` Bruce Ellis 2009-02-12 16:52 ` erik quanstrom 2009-02-10 22:27 ` Nathaniel W Filardo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).