[9fans] /n/sources/contrib/

[9fans] /n/sources/contrib/

[lucio]

I started to mirror the entirety of /n/sources/contrib, but the
results are disappointing.

If I'm not mistaken, there is no "replica" facility around "contrib",
could it be arranged?

++L

Re: [9fans] /n/sources/contrib/

[andrey mirtchovski]

that will require BL to give each author in there the ability to
"push" things to the repository.

i suggest using something like rsync (tra, most likely:
http://swtch.com/tra/) to keep track of contrib updates.

On Nov 16, 2007 10:35 PM,  <lucio@proxima.alt.za> wrote:
> I started to mirror the entirety of /n/sources/contrib, but the
> results are disappointing.
>
> If I'm not mistaken, there is no "replica" facility around "contrib",
> could it be arranged?
>
> ++L
>
>

Re: [9fans] /n/sources/contrib/

[Uriel]

http://plan9.bell-labs.com/wiki/plan9/Setting_up_a_sources_mirror/

On Nov 17, 2007 6:35 AM,  <lucio@proxima.alt.za> wrote:
> I started to mirror the entirety of /n/sources/contrib, but the
> results are disappointing.
>
> If I'm not mistaken, there is no "replica" facility around "contrib",
> could it be arranged?
>
> ++L
>
>

Re: [9fans] /n/sources/contrib/

[lucio]

> i suggest using something like rsync (tra, most likely:
> http://swtch.com/tra/) to keep track of contrib updates.

That's a reasonable idea, thank you.  I'll look into it.

++L

Re: [9fans] /n/sources/contrib/

[lucio]

> that will require BL to give each author in there the ability to
> "push" things to the repository.

Hm, I see your point, r maybe I dont.  I was thinking along the lines
of running a "push" daily or so automatically, rather.

++L

Re: [9fans] /n/sources/contrib/

[lucio]

> http://plan9.bell-labs.com/wiki/plan9/Setting_up_a_sources_mirror/

Thank you, Uriel, I'll have a good look.

++L

Re: [9fans] /n/sources/contrib/

[erik quanstrom]

On Sat Nov 17 13:40:20 EST 2007, mirtchovski@gmail.com wrote:
> that will require BL to give each author in there the ability to
> "push" things to the repository.
> 
> i suggest using something like rsync (tra, most likely:
> http://swtch.com/tra/) to keep track of contrib updates.
> 
> On Nov 16, 2007 10:35 PM,  <lucio@proxima.alt.za> wrote:
> > I started to mirror the entirety of /n/sources/contrib, but the
> > results are disappointing.
> >
> > If I'm not mistaken, there is no "replica" facility around "contrib",
> > could it be arranged?
> >
> > ++L

why not do this at a level below the fs.  venti is write once.  so
after a fossil dump, the skinny on the arena offsets and the score
of the superblock (or whatever fossil calls it) could be relayed
to the mirror site which could pull (or be pushed) the updated
arena(s).  recover (or whatever fossil calls it) the mirror fossil
from the new superblock and you're done.

- erik

Re: [9fans] /n/sources/contrib/

[lucio]

> why not do this at a level below the fs.  venti is write once.  so
> after a fossil dump, the skinny on the arena offsets and the score
> of the superblock (or whatever fossil calls it) could be relayed
> to the mirror site which could pull (or be pushed) the updated
> arena(s).  recover (or whatever fossil calls it) the mirror fossil
> from the new superblock and you're done.

That makes a lot of sense.  How practical is it?  At minimum, there
would have to be a dedicated Venti to isolate "contrib" from other
sections, plus the arenas would be large: not everyone has broadband
access.

What would be beneficial, but I understand how hard it would be to
change to it, is if contributors restricted their offering to small
source files rather than large binary objects, specially compressed
ones that probably affect the Venti archive negatively as well.  But
maybe I'm missing more than I'm aware of, I'd like to hear different
takes on this.

++L

Re: [9fans] /n/sources/contrib/

[erik quanstrom]

> That makes a lot of sense.  How practical is it?  At minimum, there
> would have to be a dedicated Venti to isolate "contrib" from other
> sections, plus the arenas would be large: not everyone has broadband
> access.

only the differences in the arenas would need to be copied.  by definition,
the differences in the arenas are not bigger than the differences in the files.
bonus: you don't have to calculate the differences in the files.

this is something i am doing with ken's fs.  i think the outline is the same
for venti.  though given superblocks n, n+1 on the fs, one knows which
data to copy as superblocks are labeled by w address (block number).
the starting lba is waddr(n)*(RBUFSIZE/512) and you need to copy
(waddr(n+1)-waddr(n))/(RBUFSIZE/512) disk blocks.

not quite sure how that translates to venti.

- erik

Re: [9fans] /n/sources/contrib/

[lucio]

> only the differences in the arenas would need to be copied.  by definition,
> the differences in the arenas are not bigger than the differences in the files.
> bonus: you don't have to calculate the differences in the files.

That's true indeed, I didn't think of that.  Hmm...

++L

Re: [9fans] /n/sources/contrib/

[johnny]

[-- Attachment #1: Type: text/plain, Size: 352 bytes --]

Maybe I'm missing something, but I would think it wouldn't be too hard to walk the directories in /n/sources/contrib and vac them. You could then store the olc vac hashes in a file, and everyday use them to copy back tou your venti server only the differences (vac -d). This could also be optimised by adding the -q option to vac.
Cheers
John Soros

[-- Attachment #2: Type: message/rfc822, Size: 2975 bytes --]

From: lucio@proxima.alt.za
To: 9fans@cse.psu.edu
Subject: Re: [9fans] /n/sources/contrib/
Date: Sun, 18 Nov 2007 07:30:43 +0200
Message-ID: <c781cd1677b5eb4339da3500f565b4ef@proxima.alt.za>

> only the differences in the arenas would need to be copied.  by definition,
> the differences in the arenas are not bigger than the differences in the files.
> bonus: you don't have to calculate the differences in the files.

That's true indeed, I didn't think of that.  Hmm...

++L

Re: [9fans] /n/sources/contrib/

[lucio]

> Maybe I'm missing something, but I would think it wouldn't be too hard
> to walk the directories in /n/sources/contrib and vac them.  You could
> then store the olc vac hashes in a file, and everyday use them to copy
> back tou your venti server only the differences (vac -d).  This could
> also be optimised by adding the -q option to vac.

I must confess that I don't quite grasp all the subtleties of Venti
(some might say without fear of contradiction that I don't really get
Venti at all :-), so the above seems perfectly reasonable to me.  This
is getting more and more interesting.

++L

Re: [9fans] /n/sources/contrib/

[roger peppe]

On Nov 17, 2007 10:00 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> after a fossil dump, the skinny on the arena offsets and the score
> of the superblock (or whatever fossil calls it) could be relayed
> to the mirror site [...]

you'd have to make sure if you did that that people putting stuff in
contrib were aware
that setting a file's permissions to 600 would have no effect on anyone's
ability to read it. i realise that it is called "contrib" but it does *look*
as if file permissions should work.

i've always wondered if there might be some way of changing venti
to make it more secure for multi-user access, so accidentally sharing
a root score
isn't quite such a terrible thing.

Re: [9fans] /n/sources/contrib/

[lucio]

> i've always wondered if there might be some way of changing venti
> to make it more secure for multi-user access, so accidentally sharing
> a root score
> isn't quite such a terrible thing.

Encryption, specially public key encryption, would do the trick, I
think.  But I confess that the details are beyond my ken.

++L

Re: [9fans] /n/sources/contrib/

[erik quanstrom]

> you'd have to make sure if you did that that people putting stuff in
> contrib were aware
> that setting a file's permissions to 600 would have no effect on anyone's
> ability to read it. i realise that it is called "contrib" but it does *look*
> as if file permissions should work.
> 
> i've always wondered if there might be some way of changing venti
> to make it more secure for multi-user access, so accidentally sharing
> a root score
> isn't quite such a terrible thing.

ignoring the fact that venti doesn't know for (fossil) users, does no
authentication ....

that's quite a radical shift from unix-like behavior.  unix permissions have
always been enforced at the file, not the block level.  this could lead to
situations where you can read parts of a directory entry but get Eperm
on other parts. or where you could get Eperm in the middle of a file.

idle questions:
· has anyone considered making venti extent rather than block based?
· does anyone have any performance numbers on resonably fast/modern
hardware for venti?

- erik

Re: [9fans] /n/sources/contrib/

[Russ Cox]

> I started to mirror the entirety of /n/sources/contrib, but the
> results are disappointing.
> 
> If I'm not mistaken, there is no "replica" facility around "contrib",
> could it be arranged?

The web page Uriel pointed at contains a recipe but no explanation.
The explanation is that every night the file /n/sources/lsr is updated
with a list of all the files on sources and their modification times,
sizes, and content hashes.  You can copy lsr and then diff it against
your previous copy of lsr to find out which files have changed 
and need to be updated in your "replica".

It's very easy.  Much easier than trying to build something like this
on top of venti or fossil, and with the added benefit that you can
selectively mirror, excluding some trees as desired.

Russ

Re: [9fans] /n/sources/contrib/

[erik quanstrom]

> The web page Uriel pointed at contains a recipe but no explanation.
> The explanation is that every night the file /n/sources/lsr is updated
> with a list of all the files on sources and their modification times,
> sizes, and content hashes.  You can copy lsr and then diff it against
> your previous copy of lsr to find out which files have changed 
> and need to be updated in your "replica".
> 
> It's very easy.  Much easier than trying to build something like this
> on top of venti or fossil, and with the added benefit that you can
> selectively mirror, excluding some trees as desired.

why would coping the changed bits of the
active arena be so hard?  is it too hard to update
the index?

with ken's fs, copying from superblockn->w-address to
superblockn+1->waddr and recover is sufficient.
the bonus is that history works.

- erik

Re: [9fans] /n/sources/contrib/

[johnny]

[-- Attachment #1: Type: text/plain, Size: 209 bytes --]

I don't understand, wouldn't 600 permission make it impossible to read to anyone but the user himself? what is the issue then, if noone can read it, vac can't read it either, or am I missing something?
John

[-- Attachment #2: Type: message/rfc822, Size: 4201 bytes --]

From: "roger peppe" <rogpeppe@gmail.com>
To: "Fans of the OS Plan 9 from Bell Labs" <9fans@cse.psu.edu>
Subject: Re: [9fans] /n/sources/contrib/
Date: Sun, 18 Nov 2007 11:33:40 +0000
Message-ID: <df49a7370711180333i109670a9k4ce5c0ce557114a1@mail.gmail.com>

On Nov 17, 2007 10:00 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> after a fossil dump, the skinny on the arena offsets and the score
> of the superblock (or whatever fossil calls it) could be relayed
> to the mirror site [...]

you'd have to make sure if you did that that people putting stuff in
contrib were aware
that setting a file's permissions to 600 would have no effect on anyone's
ability to read it. i realise that it is called "contrib" but it does *look*
as if file permissions should work.

i've always wondered if there might be some way of changing venti
to make it more secure for multi-user access, so accidentally sharing
a root score
isn't quite such a terrible thing.

Re: [9fans] /n/sources/contrib/

[Francisco J Ballesteros]

It's not hard.
We have been doing that to keep a local "mirror" of our main
file server (both use venti/fossil)

On 11/18/07, erik quanstrom <quanstro@coraid.com> wrote:
> why would coping the changed bits of the
> active arena be so hard?  is it too hard to update
> the index?
>
> with ken's fs, copying from superblockn->w-address to
> superblockn+1->waddr and recover is sufficient.
> the bonus is that history works.
>
> - erik
>

Re: [9fans] /n/sources/contrib/

[johnny]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]

I might be missing something again, but if someone set's a 600 permission, doen't that mean only the user himself can see the file? I don't understand why this is relevant, as if that's the case, noone else can see the file. The only issue is if the user who does the vac backup backs up his own contrib directory logged in as himself.
As for backing up stand alone directories and files, doesn't vac do that just fine?
Cheers!
Johnny

[-- Attachment #2: Type: message/rfc822, Size: 4201 bytes --]

From: "roger peppe" <rogpeppe@gmail.com>
To: "Fans of the OS Plan 9 from Bell Labs" <9fans@cse.psu.edu>
Subject: Re: [9fans] /n/sources/contrib/
Date: Sun, 18 Nov 2007 11:33:40 +0000
Message-ID: <df49a7370711180333i109670a9k4ce5c0ce557114a1@mail.gmail.com>

On Nov 17, 2007 10:00 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> after a fossil dump, the skinny on the arena offsets and the score
> of the superblock (or whatever fossil calls it) could be relayed
> to the mirror site [...]

you'd have to make sure if you did that that people putting stuff in
contrib were aware
that setting a file's permissions to 600 would have no effect on anyone's
ability to read it. i realise that it is called "contrib" but it does *look*
as if file permissions should work.

i've always wondered if there might be some way of changing venti
to make it more secure for multi-user access, so accidentally sharing
a root score
isn't quite such a terrible thing.

Re: [9fans] /n/sources/contrib/

[Russ Cox]

>> The web page Uriel pointed at contains a recipe but no explanation.
>> The explanation is that every night the file /n/sources/lsr is updated
>> with a list of all the files on sources and their modification times,
>> sizes, and content hashes.  You can copy lsr and then diff it against
>> your previous copy of lsr to find out which files have changed 
>> and need to be updated in your "replica".
>> 
>> It's very easy.  Much easier than trying to build something like this
>> on top of venti or fossil, and with the added benefit that you can
>> selectively mirror, excluding some trees as desired.
> 
> why would coping the changed bits of the
> active arena be so hard?  is it too hard to update
> the index?
> 
> with ken's fs, copying from superblockn->w-address to
> superblockn+1->waddr and recover is sufficient.
> the bonus is that history works.

it's not necessarily hard -- you just write the code -- but it's
more fragile to do this kind of exact mirroring, because you
can't make any local modifications without breaking the 
mirroring.  if you decide you don't want to keep some huge
subtree, for example, or you want to add a few locally
maintained files.  

the file system is a good interface -- witness all of plan 9 --
and if you're going to start working at a lower level i just
think you need a much more compelling reason.

if your goal is to have a live backup of a system, then some
disk-level thing might be exactly right.  but for having a 
local mirror of sources, the lsr file and a simple script to
run the cp commands is simple and more flexible.

russ

Re: [9fans] /n/sources/contrib/

[roger peppe]

On Nov 18, 2007 6:17 PM,  <johnny@sorosj.hd.free.fr> wrote:
> I don't understand, wouldn't 600 permission make it impossible to read to
> anyone but the user himself? what is the issue then, if noone can read it,
> vac can't read it either, or am I missing something?

i think that erik was talking about conveying the root score
and having clients talk to the venti server directly, rather
than through fossil. in this case, once you've got the root
score, you've got the whole thing - venti doesn't know about
file permissions. using vac is a much better idea (it'd be even
better if directory mtimes reflected the max of the mtimes of any of their
descendants. but the ls -lt manifest removes most of that inefficiency)

[BTW, has anyone else been having a very slow response time from the list?
i only just saw that reply and it was sent 22 hours ago]

Re: [9fans] /n/sources/contrib/

[erik quanstrom]

> it's not necessarily hard -- you just write the code -- but it's
> more fragile to do this kind of exact mirroring, because you
> can't make any local modifications without breaking the 
> mirroring.  if you decide you don't want to keep some huge
> subtree, for example, or you want to add a few locally
> maintained files.

on the other hand, a mirror doesn't really operate at the
level of the filesystem.  a mirror needs to avoid the normal
permission checking. my experience has been that filesystem-level
copies tend to have errors in them and the errors increase over
time.  and the muid is wrong.  that's a big deal for us.

maybe plan 9 needs dump.

i could be out of step here.  my basic assumption is that unless
a mirror is exact, it's not a mirror.  you know, might be missing
some files.  owners might be different.  muid broke. you know, ymmv.
if i set up mirrors like that, ever time anyone had a problem
with a file, they'd blame me first.

> the file system is a good interface -- witness all of plan 9 --
> and if you're going to start working at a lower level i just
> think you need a much more compelling reason.

the file interface has nothing to do with this problem.
it's the arenas that need copying.  if you do that, the files will
take care of themselves.

venti itself doesn't even use 9p.

> if your goal is to have a live backup of a system, then some
> disk-level thing might be exactly right.  but for having a 
> local mirror of sources, the lsr file and a simple script to
> run the cp commands is simple and more flexible.

by the way, if the datestamp in the lsr file is not the current
day (you know, cp -x), you'll miss the file with the lsr file.

- erik

Re: [9fans] /n/sources/contrib/

[Russ Cox]

> by the way, if the datestamp in the lsr file is not the current
> day (you know, cp -x), you'll miss the file with the lsr file.

no, you won't.  i said to diff the two lsr files and copy
the ones whose lines had changed.  so any change to
the modification time, size, or content hash will cause
you to recopy the file.

> i could be out of step here.  my basic assumption is that unless
> a mirror is exact, it's not a mirror. 

that is a fine definition of mirror.  but other people use other
definitions.  i might be willing to mirror /n/sources/plan9
but not /n/sources/contrib, for example.  or i might want to
mirror all of sources as well as some other servers all onto
one big server.  (think the big ftp mirrors.)

as i said, if you are interested in your definition of mirror,
copying the low-level bits works great.  if you want to be
a little more flexible, then that approach stops being appropriate.

russ

[9fans] /n/sources/contrib/

[David Hoskin]

How does one sign up for a contrib directory these days?

I tried the addresses in
http://www.plan9.bell-labs.com/wiki/plan9/how_to_contribute/
http://9fans.net/archive/2013/05/231
a few months ago, but never heard anything back.

-- David