[9fans] Sources Gone?

[9fans] Sources Gone?

[Gregory Pavelcak]

I usually try to avoid the "what happened to sources" email,
but what happened to sources? All I've been getting from
pull attempts for the past several days is

srv: dial tcp!sources.cs.bell-labs.com!9fs: connection refused
bind: /n/sources/plan9: '/n/sources/plan9' does not exist
servermount: bind 545911: bind

Could there be something wrong at my end?

Thanks.

Greg

Re: [9fans] Sources Gone?

[erik quanstrom]

> srv: dial tcp!sources.cs.bell-labs.com!9fs: connection refused
> bind: /n/sources/plan9: '/n/sources/plan9' does not exist
> servermount: bind 545911: bind

the problem is on your end.  sounds like
either a dns problem or port blocking.

- erik

Re: [9fans] Sources Gone?

[lucio]

> srv: dial tcp!sources.cs.bell-labs.com!9fs: connection refused
> bind: /n/sources/plan9: '/n/sources/plan9' does not exist
> servermount: bind 545911: bind

Hm, about as bad as when a couple of days ago I left replica/pull
running overnight and came back next morning to find that even /bin/ls
had disappeared.

Maybe sources _is_ sick.

++L

PS: recovering from fossil/venti was easy, although I had to hunt down
the previous day's score (thank you Russ for "vacchain", warts and
all), but my attempts at fixing the problem by less disruptive means
were quite an experience.

Re: [9fans] Sources Gone?

[erik quanstrom]

> > srv: dial tcp!sources.cs.bell-labs.com!9fs: connection refused
> > bind: /n/sources/plan9: '/n/sources/plan9' does not exist
> > servermount: bind 545911: bind
>
> Hm, about as bad as when a couple of days ago I left replica/pull
> running overnight and came back next morning to find that even /bin/ls
> had disappeared.
>
> Maybe sources _is_ sick.

it's known that replica can do some antisocial things.
i don't think it's necessary to invoke mystery problems
to explain a replica's issues.

- erik

Re: [9fans] Sources Gone?

[Uriel]

On Fri, Jan 23, 2009 at 3:54 PM,  <lucio@proxima.alt.za> wrote:
> Hm, about as bad as when a couple of days ago I left replica/pull
> running overnight and came back next morning to find that even /bin/ls
> had disappeared.

You are not the only one to have woken up with such pleasant surprise
several times.

> Maybe sources _is_ sick.

Or maybe replica sucks.

But people keep telling me that replica's unreliability, painful
slowness, and general clunkyness,  are all in my imagination, so what
do I know...

uriel

Re: [9fans] Sources Gone?

[Patrick Kristiansen]

[-- Attachment #1: Type: text/plain, Size: 965 bytes --]

I did a pull yesterday and it has removed all my /386/ and it is now unable
to boot.

boot: /386/init: '/386/init' does not exist
panic: boot process died: unknown
panic: boot process died: unknown
dumpstack disabled
cpu0: exiting


Hints on how to restore are welcome....

-Patrick

2009/1/23 <lucio@proxima.alt.za>

> > srv: dial tcp!sources.cs.bell-labs.com!9fs: connection refused
> > bind: /n/sources/plan9: '/n/sources/plan9' does not exist
> > servermount: bind 545911: bind
>
> Hm, about as bad as when a couple of days ago I left replica/pull
> running overnight and came back next morning to find that even /bin/ls
> had disappeared.
>
> Maybe sources _is_ sick.
>
> ++L
>
> PS: recovering from fossil/venti was easy, although I had to hunt down
> the previous day's score (thank you Russ for "vacchain", warts and
> all), but my attempts at fixing the problem by less disruptive means
> were quite an experience.
>
>
>

[-- Attachment #2: Type: text/html, Size: 1413 bytes --]

Re: [9fans] Sources Gone?

[Russ Cox]

> But people keep telling me that replica's unreliability, painful
> slowness, and general clunkyness,  are all in my imagination, so what
> do I know...

No, what we've told you, repeatedly, is that
whining about problems and fixing them are
two different things.  Fixes are appreciated.

Russ

Re: [9fans] Sources Gone?

[Tharaneedharan Vilwanathan]

[-- Attachment #1: Type: text/plain, Size: 1615 bytes --]

hi patrick,

someone else can give a better answer for sure but let me give give a try
here. (sorry if you know this already or if i have misunderstood your
question).

would reverting to an older fossil snapshot work for you?

if so, you could do this (i have just outlined):

- boot via cdrom
- find fossil's last score (using fossil/last command)
- format fossil partition with that score (flfmt command)

now, remove cdrom and boot from hard disk.

hope this helps.

thanks
dharani

On Tue, Jan 27, 2009 at 3:11 PM, Patrick Kristiansen <
patrick.kasserer@gmail.com> wrote:

> I did a pull yesterday and it has removed all my /386/ and it is now unable
> to boot.
>
> boot: /386/init: '/386/init' does not exist
> panic: boot process died: unknown
> panic: boot process died: unknown
> dumpstack disabled
> cpu0: exiting
>
>
> Hints on how to restore are welcome....
>
> -Patrick
>
> 2009/1/23 <lucio@proxima.alt.za>
>
>  > srv: dial tcp!sources.cs.bell-labs.com!9fs: connection refused
>> > bind: /n/sources/plan9: '/n/sources/plan9' does not exist
>> > servermount: bind 545911: bind
>>
>> Hm, about as bad as when a couple of days ago I left replica/pull
>> running overnight and came back next morning to find that even /bin/ls
>> had disappeared.
>>
>> Maybe sources _is_ sick.
>>
>> ++L
>>
>> PS: recovering from fossil/venti was easy, although I had to hunt down
>> the previous day's score (thank you Russ for "vacchain", warts and
>> all), but my attempts at fixing the problem by less disruptive means
>> were quite an experience.
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 2527 bytes --]

Re: [9fans] Sources Gone?

[Kenji Arisawa]

Hello,

Several years ago I abandoned replica and switched to my own tool
"upadate",
but I hesitated to make it public because replica is so fundamental
tools
for Plan 9.
The major problem is(was?) replica doesn't properly arrange update
data-base
before it begins to retrieve files. The laxness makes replica very slow.

If replica is to be improved I hope replica looks owner/group
information of files
in updating. If these information is different from official one, then
the file should be
regarded as having modified by users.

Kenji Arisawa

On 2009/01/28, at 8:32, Russ Cox wrote:

>> But people keep telling me that replica's unreliability, painful
>> slowness, and general clunkyness,  are all in my imagination, so what
>> do I know...
>
> No, what we've told you, repeatedly, is that
> whining about problems and fixing them are
> two different things.  Fixes are appreciated.
>
> Russ
>

Re: [9fans] Sources Gone?

[Uriel]

Mercurial and git solve all replica problems, and some more.

They are infinitely faster, more reliable, and more useful. And in
some ways they are even conceptually simpler (I never quite understood
some of the most subtle points of replica, like why it keeps saying it
needs to update files that were already updated if I happen to have
some local changes elsewhere, even when I have had them explained to
me repeatedly, of course that is due to my own intellectual
limitations, but...)

The git codebase is considerably more complex, but that is because it
does tons of things we don't really need to replace replica, I even
got a port of git almost going. If it is agreed to replace replica
with git, I will be happy to work on porting the latest git version.
(It already works under linuxemu).

Mercurial is much simpler (at some point it was about seven thousand
lines of python, it has grown since, but the useful stuff was all
already there), and it think some people got it to work with their
various python ports (not sure which, there are so many that I lost
track), and mjl has a very nice read only limbo implementation, and
said making it write wouldn't be too hard (the tricky stuff is merges
and such, which we don't really need to replace replica).

So, all that is needed to get rid of replica is the determination to
do so, and I'm sure many people will be happy to help in the effort.

Peace

uriel

P.S.: Other problems this would solve is fast and efficient mirroring,
web interfaces to the source and history, etc.

On Wed, Jan 28, 2009 at 12:32 AM, Russ Cox <rsc@swtch.com> wrote:
>> But people keep telling me that replica's unreliability, painful
>> slowness, and general clunkyness,  are all in my imagination, so what
>> do I know...
>
> No, what we've told you, repeatedly, is that
> whining about problems and fixing them are
> two different things.  Fixes are appreciated.
>
> Russ
>
>

Re: [9fans] Sources Gone?

[lucio]

> - find fossil's last score (using fossil/last command)
> - format fossil partition with that score (flfmt command)

It may be necessary to go back one score, the morning surprise tends
to be after the archive snap.

I had to hack "vacchain" to return the score audit trail.  Ask Russ
for the original, the venti/read now no longer needs (or accepts) the
trailing "1" block count.

++L

Re: [9fans] Sources Gone?

[Iruata Souza]

On Wed, Jan 28, 2009 at 3:06 AM, Uriel <uriel99@gmail.com> wrote:
> Mercurial and git solve all replica problems, and some more.
>
> They are infinitely faster, more reliable, and more useful. And in
> some ways they are even conceptually simpler (I never quite understood
> some of the most subtle points of replica, like why it keeps saying it
> needs to update files that were already updated if I happen to have
> some local changes elsewhere, even when I have had them explained to
> me repeatedly, of course that is due to my own intellectual
> limitations, but...)
>
> The git codebase is considerably more complex, but that is because it
> does tons of things we don't really need to replace replica, I even
> got a port of git almost going. If it is agreed to replace replica
> with git, I will be happy to work on porting the latest git version.
> (It already works under linuxemu).
>
> Mercurial is much simpler (at some point it was about seven thousand
> lines of python, it has grown since, but the useful stuff was all
> already there), and it think some people got it to work with their
> various python ports (not sure which, there are so many that I lost
> track), and mjl has a very nice read only limbo implementation, and
> said making it write wouldn't be too hard (the tricky stuff is merges
> and such, which we don't really need to replace replica).
>
> So, all that is needed to get rid of replica is the determination to
> do so, and I'm sure many people will be happy to help in the effort.
>
> Peace
>
> uriel
>
> P.S.: Other problems this would solve is fast and efficient mirroring,
> web interfaces to the source and history, etc.
>
> On Wed, Jan 28, 2009 at 12:32 AM, Russ Cox <rsc@swtch.com> wrote:
>>> But people keep telling me that replica's unreliability, painful
>>> slowness, and general clunkyness,  are all in my imagination, so what
>>> do I know...
>>
>> No, what we've told you, repeatedly, is that
>> whining about problems and fixing them are
>> two different things.  Fixes are appreciated.
>>
>> Russ
>>
>>
>
>

contrib/bichued has a working mercurial port.

iru

Re: [9fans] Sources Gone?

[Charles Forsyth]

before getting too excited replacing one thing by another,
it's always worth discovering what actually did go wrong,
just in case the other thing would (for instance) have done much
the same thing in the circumstances, or perhaps has
different drawbacks.  perhaps the problem is in the
script that updates the log. i don't know, but i'd certainly
find out if i could.

Re: [9fans] Sources Gone?

[erik quanstrom]

> They [hg/mecurial] are infinitely faster, more reliable, and more useful. And in
> some ways they are even conceptually simpler (I never quite understood
> some of the most subtle points of replica, like why it keeps saying it
> needs to update files that were already updated if I happen to have
> some local changes elsewhere, even when I have had them explained to
> me repeatedly, of course that is due to my own intellectual
> limitations, but...)

charles hit the main point — it's hard to fix things without
identifying the problem.

i do not use replica to update my machine, but this is a reflection
of the fact that legitimate changes to files i've never changed can
remove functionality that i use, like il.  i suspect that i'm a special
case in this regard and no amount of revision control fanciness
can save me.

i have used replica to move history between fs with different
on-disk layouts.  http://www.quanstro.net/plan9/history.pdf
has all the gory details.  i've been able to successfully
replicate about 3000 filesystems without a single problem.
but i had pretty controlled circumstances and i didn't have
any fs trouble.

in the case of sources, i think something is going wrong while
updatedb is running.  (hg or mecurial would likely suffer the same
fate, btw.)  from the log on sources:

; grep 386/init plan9.log
1196890780 540 a 386/init - 775 sys sys 1179372116 101487
1209616216 98 c 386/init - 775 sys sys 1209614871 101498
1219773605 695 a 386/init - 775 sys sys 1209614871 101498
1232008203 15347 d 386/init - 775 sys sys 1209614871 0
1232038858 697 a 386/init - 775 sys sys 1209614871 101498

the 3d column is the action 'a' for add 'c' for change and 'd'
for delete.  notice that other than the first and second line,
all the actions are suprising.

since several people have reported venti errors when
connecting to sources recently, it seems to me that the
most logical explination is that when the update process
ran on 15 jan, venti was mia.  then replica interpreted
the error to mean that 386/init had been deleted.

supposing this is correct, it would be logical to add this
patch which i added to replica for the history paper.
i think that adding a strcmp for "venti error"
to this patch would do the trick.

/n/sources/plan9//sys/src/cmd/replica/updatedb.c:77,87 - updatedb.c:77,91
  			change = 1;
  		}
  	}else{
- 		if((d.mode&DMDIR)==0 && (od.mtime!=d.mtime || od.length!=d.length)){
+ 		if((od.mode&DMDIR) != (d.mode&DMDIR)){
+ 			xlog('d', new, &d);
+ 			xlog('a', new, &d);
+ 			change = 1;
+ 		}else if((d.mode&DMDIR)==0 && (od.mtime!=d.mtime || od.length!=d.length)){
  			xlog('c', new, &d);
  			change = 1;
  		}
- 		if((!uid&&strcmp(od.uid,d.uid)!=0)
+ 		else if((!uid&&strcmp(od.uid,d.uid)!=0)
  		|| strcmp(od.gid,d.gid)!=0
  		|| od.mode!=d.mode){
  			xlog('m', new, &d);
/n/sources/plan9//sys/src/cmd/replica/updatedb.c:97,135 - updatedb.c:101,106
  }

  void
- warn(char *msg, void*)
- {
- 	char *p;
-
- 	fprint(2, "warning: %s\n", msg);
-
- 	/* find the %r in "can't open foo: %r" */
- 	p = strstr(msg, ": ");
- 	if(p)
- 		p += 2;
-
- 	/*
- 	 * if the error is about a remote server failing,
- 	 * then there's no point in continuing to look
- 	 * for changes -- we'll think everything got deleted!
- 	 *
- 	 * actual errors i see are:
- 	 *	"i/o on hungup channel" for a local hangup
- 	 *	"i/o on hungup channel" for a timeout (yank the network wire)
- 	 *	"'/n/sources/plan9' Hangup" for a remote hangup
- 	 * the rest is paranoia.
- 	 */
- 	if(p){
- 		if(cistrstr(p, "hungup") || cistrstr(p, "Hangup")
- 		|| cistrstr(p, "rpc error")
- 		|| cistrstr(p, "shut down")
- 		|| cistrstr(p, "i/o")
- 		|| cistrstr(p, "connection"))
- 			sysfatal("suspected network or i/o error - bailing out");
- 	}
- }
-
- void
  usage(void)
  {
  	fprint(2, "usage: replica/updatedb [-c] [-p proto] [-r root] [-t now n] [-u uid] [-x path]... db [paths]\n");
/n/sources/plan9//sys/src/cmd/replica/updatedb.c:184,190 - updatedb.c:155,161
  	nmatch = argc-1;

  	db = opendb(argv[0]);
- 	if(rdproto(proto, root, walk, warn, nil) < 0)
+ 	if(rdproto(proto, root, walk, nil, nil) < 0)
  		sysfatal("rdproto: %r");

  	if(!changesonly){


- erik

Re: [9fans] Sources Gone?

[Uriel]

The issues with replica go beyond its tendency to wipe out complete
file systems.

It includes, among other things, the performance of a drunken slug,
and as you well point out, the skils of a schizophrenic monkey for
managing local changes.

All this has been solved by git and hg; and git and hg would *never*
wipe out your local files simply because the backing store for the
repository you are pulling from happens to break, the pull simply
would fail and leave your local repo intact, and when the remote repo
is brought back, all would work just fine.

Oh, and they are both *excellent* at helping one keep track of local
changes without messing everything up.

Of course they also help with things like merges, changelog
generation, etc. but I suspect those things are not really wanted.

uriel


On Wed, Jan 28, 2009 at 2:53 PM, erik quanstrom <quanstro@quanstro.net> wrote:
>> They [hg/mecurial] are infinitely faster, more reliable, and more useful. And in
>> some ways they are even conceptually simpler (I never quite understood
>> some of the most subtle points of replica, like why it keeps saying it
>> needs to update files that were already updated if I happen to have
>> some local changes elsewhere, even when I have had them explained to
>> me repeatedly, of course that is due to my own intellectual
>> limitations, but...)
>
> charles hit the main point — it's hard to fix things without
> identifying the problem.
>
> i do not use replica to update my machine, but this is a reflection
> of the fact that legitimate changes to files i've never changed can
> remove functionality that i use, like il.  i suspect that i'm a special
> case in this regard and no amount of revision control fanciness
> can save me.
>
> i have used replica to move history between fs with different
> on-disk layouts.  http://www.quanstro.net/plan9/history.pdf
> has all the gory details.  i've been able to successfully
> replicate about 3000 filesystems without a single problem.
> but i had pretty controlled circumstances and i didn't have
> any fs trouble.
>
> in the case of sources, i think something is going wrong while
> updatedb is running.  (hg or mecurial would likely suffer the same
> fate, btw.)  from the log on sources:
>
> ; grep 386/init plan9.log
> 1196890780 540 a 386/init - 775 sys sys 1179372116 101487
> 1209616216 98 c 386/init - 775 sys sys 1209614871 101498
> 1219773605 695 a 386/init - 775 sys sys 1209614871 101498
> 1232008203 15347 d 386/init - 775 sys sys 1209614871 0
> 1232038858 697 a 386/init - 775 sys sys 1209614871 101498
>
> the 3d column is the action 'a' for add 'c' for change and 'd'
> for delete.  notice that other than the first and second line,
> all the actions are suprising.
>
> since several people have reported venti errors when
> connecting to sources recently, it seems to me that the
> most logical explination is that when the update process
> ran on 15 jan, venti was mia.  then replica interpreted
> the error to mean that 386/init had been deleted.
>
> supposing this is correct, it would be logical to add this
> patch which i added to replica for the history paper.
> i think that adding a strcmp for "venti error"
> to this patch would do the trick.
>
> /n/sources/plan9//sys/src/cmd/replica/updatedb.c:77,87 - updatedb.c:77,91
>                        change = 1;
>                }
>        }else{
> -               if((d.mode&DMDIR)==0 && (od.mtime!=d.mtime || od.length!=d.length)){
> +               if((od.mode&DMDIR) != (d.mode&DMDIR)){
> +                       xlog('d', new, &d);
> +                       xlog('a', new, &d);
> +                       change = 1;
> +               }else if((d.mode&DMDIR)==0 && (od.mtime!=d.mtime || od.length!=d.length)){
>                        xlog('c', new, &d);
>                        change = 1;
>                }
> -               if((!uid&&strcmp(od.uid,d.uid)!=0)
> +               else if((!uid&&strcmp(od.uid,d.uid)!=0)
>                || strcmp(od.gid,d.gid)!=0
>                || od.mode!=d.mode){
>                        xlog('m', new, &d);
> /n/sources/plan9//sys/src/cmd/replica/updatedb.c:97,135 - updatedb.c:101,106
>  }
>
>  void
> - warn(char *msg, void*)
> - {
> -       char *p;
> -
> -       fprint(2, "warning: %s\n", msg);
> -
> -       /* find the %r in "can't open foo: %r" */
> -       p = strstr(msg, ": ");
> -       if(p)
> -               p += 2;
> -
> -       /*
> -        * if the error is about a remote server failing,
> -        * then there's no point in continuing to look
> -        * for changes -- we'll think everything got deleted!
> -        *
> -        * actual errors i see are:
> -        *      "i/o on hungup channel" for a local hangup
> -        *      "i/o on hungup channel" for a timeout (yank the network wire)
> -        *      "'/n/sources/plan9' Hangup" for a remote hangup
> -        * the rest is paranoia.
> -        */
> -       if(p){
> -               if(cistrstr(p, "hungup") || cistrstr(p, "Hangup")
> -               || cistrstr(p, "rpc error")
> -               || cistrstr(p, "shut down")
> -               || cistrstr(p, "i/o")
> -               || cistrstr(p, "connection"))
> -                       sysfatal("suspected network or i/o error - bailing out");
> -       }
> - }
> -
> - void
>  usage(void)
>  {
>        fprint(2, "usage: replica/updatedb [-c] [-p proto] [-r root] [-t now n] [-u uid] [-x path]... db [paths]\n");
> /n/sources/plan9//sys/src/cmd/replica/updatedb.c:184,190 - updatedb.c:155,161
>        nmatch = argc-1;
>
>        db = opendb(argv[0]);
> -       if(rdproto(proto, root, walk, warn, nil) < 0)
> +       if(rdproto(proto, root, walk, nil, nil) < 0)
>                sysfatal("rdproto: %r");
>
>        if(!changesonly){
>
>
> - erik
>
>

Re: [9fans] Sources Gone?

[kokamoto]

> i do not use replica to update my machine, but this is a reflection
> of the fact that legitimate changes to files i've never changed can
> remove functionality that i use, like il.  i suspect that i'm a special
> case in this regard and no amount of revision control fanciness
> can save me.

I'm second.
Ken's file server is the most reliable fileserver for Plan 9, and it's
the most conceptually right thing to describe what is Plan 9.

Kenji

Re: [9fans] Sources Gone?

[erik quanstrom]

On Thu Jan 29 07:13:32 EST 2009, uriel99@gmail.com wrote:
> It includes, among other things, the performance of a drunken slug,

i don't know how slow a drunken slug is.  but before rushing
off to replace replica, it would be useful to see where the time
is going.  you may find that your proposed replacements suffer
from the same problems for the same reasons.  in fact, i suspect
that if things are slow, it is likely due to 9p latency from sources.
the same problem would apply to any tool using the high-latency
link.

i've had no problems with the performance of the replica tools.
when using replica to replicate fileservers, i have found that
the performance of replica was very good and seems to increase
fairly linearly with the speed of the source and target.

> and as you well point out, the skils of a schizophrenic monkey for
> managing local changes.

well then, please show me how hg/git or whatever would save
me from the situation outlined.  how would hg/git know that
i was really using some code which i had never locally modified
and was then removed on sources?

if you have any specific problems, i'm sure they could be quickly
fixed.

> All this has been solved by git and hg; and git and hg would *never*
> wipe out your local files simply because the backing store for the
> repository you are pulling from happens to break, the pull simply
> would fail and leave your local repo intact, and when the remote repo
> is brought back, all would work just fine.

i'm not convinced by this argument.  could you demonstrate how hg/git
would magicly apply correct deletions and not apply erroneous
ones and demonstrate how it's faster on the plan 9 codebase.

if you believe this is the way, then put your money where your
mouth is and do the work and show us how it's better.

otherwise, there's just no point in talking about it.

- erik

Re: [9fans] Sources Gone?

[ron minnich]

On Thu, Jan 29, 2009 at 4:12 AM, Uriel <uriel99@gmail.com> wrote:

> All this has been solved by git and hg; and git and hg would *never*
> wipe out your local files simply because the backing store for the
> repository you are pulling from happens to break,

Have you used git much? Sure, it's nice. Have you tried it when it
hits 6 gbyte size as we have here, and repo is filled with binaries,
not just source?

It gets rather slow. At least for us. Also, as has been pointed out,
it is not unsurprisingly optimized to Linux. MacOS users complain to
me that it doesn't do well on MacOS.

Finally, the tools!

git                      git-get-tar-commit-id    git-read-tree
git-add                  git-grep                 git-rebase
git-add--interactive     git-gui                  git-rebase--interactive
git-am                   git-hash-object          git-receive-pack
git-annotate             git-http-fetch           git-reflog
git-apply                git-http-push            git-relink
git-archimport           git-imap-send            git-remote
git-archive              git-index-pack           git-repack
git-bisect               git-init                 git-repo-config
git-blame                git-init-db              git-request-pull
git-branch               git-instaweb             git-rerere
git-bundle               gitk                     git-reset
git-cat-file             git-log                  git-revert
git-check-attr           git-lost-found           git-rev-list
git-checkout             git-ls-files             git-rev-parse
git-checkout-index       git-ls-remote            git-rev-tree
git-check-ref-format     git-ls-tree              git-rm
git-cherry               git-mailinfo             git-send-email
git-cherry-pick          git-mailsplit            git-send-pack
git-citool               git-merge                git-shell
git-clean                git-merge-base           git-shortlog
git-clone                git-merge-file           git-show
git-commit               git-merge-index          git-show-branch

And it all seemed so simple in the early days.

Why not just understand and fix replica, instead of throwing stones?
What's the point? To replace actual coding with verbiage?

The part that confuses me most is that, IIRC, you were extolling the
virtues of venti a few days ago (with which I agree), particularly
that it is "rock solid" (true). But the current problem was a venti
outage. Conclusion: stuff happens. When stuff happens, bugs are
exposed. Bugs can be fixed. Erik's already proposed one fix, let's get
more.

ron

Re: [9fans] Sources Gone?

[Roman V. Shaposhnik]

On Thu, 2009-01-29 at 13:12 +0100, Uriel wrote:
> The issues with replica go beyond its tendency to wipe out complete
> file systems.
>
> It includes, among other things, the performance of a drunken slug,
> and as you well point out, the skils of a schizophrenic monkey for
> managing local changes.

Full disclosure: I don't use replica. But...

> All this has been solved by git and hg; and git and hg would *never*
> wipe out your local files simply because the backing store for the
> repository you are pulling from happens to break, the pull simply
> would fail and leave your local repo intact, and when the remote repo
> is brought back, all would work just fine.

...I'm really somewhat of a Git buff. So I'm obviously interested
in this discussion.

As was pointed out in a different thread Git architecture is, in fact,
quite close to venti/fossil. Venti is the immutable hash addressed
history and fossil is the index. Thus the way Git transfers history
between the repositories is conceptually very similar to transferring
venti blocks that can be reached from a designated set of VtRoots.

This has a number of useful properties in Git: the state of the history
repository has nothing to do with the state of your local write
buffer (index). You may fetch quite a few additional blocks of history,
but that doesn't force you to "reformat" your write buffer. In fact,
it is usually a good idea to: fetch, inspect and only then do
a "merge".

replica, on the other hand, tries to not rely on anything but its own
functionality, effectively making it possible to manage changes backed
by something other than fossil/venti. Replica is trying to be more
like rsync, than Git.

I've come to realize that this might be, in fact, the key problem.

Git itself is not afraid to admit that it *is* a filesystem. Linus
said it explicitly on quite a few occasions. We already have a
filesystem. Do we need another one?

> Oh, and they are both *excellent* at helping one keep track of local
> changes without messing everything up.
>
> Of course they also help with things like merges, changelog
> generation, etc. but I suspect those things are not really wanted.

And that is a complementary problem to all of the above: your history
is as good (and as helpful in doing merges, etc.) as is the effort
put into creating it in the first place. A conversation on this list
a month or so ago completely convinced me that those who make changes
to Plan9 sources see very little value in maintaining history that way.

Thanks,
Roman.

Re: [9fans] Sources Gone?

[Roman V. Shaposhnik]

On Thu, 2009-01-29 at 08:15 -0800, ron minnich wrote:
> On Thu, Jan 29, 2009 at 4:12 AM, Uriel <uriel99@gmail.com> wrote:
>
> > All this has been solved by git and hg; and git and hg would *never*
> > wipe out your local files simply because the backing store for the
> > repository you are pulling from happens to break,
>
> Have you used git much?

I do ;-)

> Sure, it's nice. Have you tried it when it
> hits 6 gbyte size as we have here, and repo is filled with binaries,
> not just source?

We hit exactly those very problems when trying to evaluate Git
for a huge internal project with a history going back to '89. We found
very attractive solutions in Git (but not in Mercurial or Bazaar at that
time) for all the issues you've identified. YMMV, of course, but I can
share more of my experience off the list, since the subject has
absolutely nothing to do with Plan9.

Thanks,
Roman.

P.S. In all fairness, the evaluation happened about a year ago. I hear
that Mercurial has improved since then. It is also possible that Git
deteriorated, although I doubt it.

Re: [9fans] Sources Gone?

[Roman V. Shaposhnik]

On Thu, 2009-01-29 at 08:37 -0500, erik quanstrom wrote:
> > and as you well point out, the skils of a schizophrenic monkey for
> > managing local changes.
>
> well then, please show me how hg/git or whatever would save
> me from the situation outlined.  how would hg/git know that
> i was really using some code which i had never locally modified
> and was then removed on sources?

it wouldn't. but the fact that it encourages a three step process:
   1. get the immutable history (whatever it is) but don't modify your
   write buffer
   2. inspect history. Git offers quite a few nice tools to manage your
   local changes in the interim, but it is conceptually similar to
   formatting an extra fossil buffer with a score corresponding to
   the "tip" of the history and simply comparing it to what you have.
   3. only when you are absolutely certain, you combine your local
   changes with whatever history brought you, then you commit and
   get the new score
makes it far less dangerous. With replica (on those two or three
occasions that I used it) it seemed that your only option is
to "hope for the best". It doesn't manage history. It manages
your write buffer.

Thanks,
Roman.

Re: [9fans] Sources Gone?

[Russ Cox]

Replica and cvs/git/mercurial/darcs/whatever solve
similar but different problems.

Onr fundamental difference is that the latter set is
intended to keep trees exactly in sync, whereas
the design of replica expects you to want some files
to contain local changes that persist and are not
synced back in the other direction.  This is anathema
to the version control systems I've used.  (Mercurial,
for instance, used to support it okay, but slowly the
"-f" flags have been dropping off commands, making it
harder and harder.)  I am acutely aware of this problem
because I use cvs and mercurial (git was far too much
trouble a few years ago; maybe it is better now) as a
clumsy synchronization mechanism for plan9port,
and the way I use these tools is the way I used replica,
not the way they are intended to be used.

Replica also attempts to reuse the functionality of
having a network mounted file system (sources)
rather than invent its own protocols, and so on.
Basically it tries to fit into Plan 9 well.
This is probably one of the reasons for the lack of
speed, but it's not insurmountable.  The addition
of fcp made it quite a bit faster, if I recall, and I'm
sure there is still more that could be done.

The replica tools are not SHA1-based because they
cannot depend on the user having a venti to manage
the blocks, and managing a separate copy of the data
(like the dvcs's do) seemed out of character with
fitting well into the surrounding system.

I can easily believe that the replica tools might
accidentally delete your whole file system (but only
the files that you hadn't changed) if sources all of
a sudden claims that the files are gone, like it did
a few days ago.  It's trying to stay in sync with
sources, after all.  This was a case that I would
never have imagined, and it probably needs some
sanity checking.

If I were working on replica today, I would probably
change it to only perform actions listed in the log,
and not take the log merely as a set of hints about
what actions to perform.  Right now, if the log says
"there's a new copy of devmnt.8" and replica looks
and sources and doesn't see it, it assumes that
there is another change to devmnt.8 coming up,
namely its deletion.  It probably shouldn't do that,
as failure modes like sources forgetting its files
demonstrate.

I wrote replica and some fraction of fossil in my spare
time, because I was using Plan 9 and wanted to
make it a more hospitable environment.  I'm not using
Plan 9 anymore, hence not maintaining those tools.
If you, as a user of Plan 9, want the tools to be
better, that's what open source is all about.  You have
the source: fix the things that matter to you and
contribute the changes back.

Russ

P. S.  Erik is right.  Understand the problem before
you throw away the software.  http://www.jwz.org/doc/cadt.html

Re: [9fans] Sources Gone?

[erik quanstrom]

> I can easily believe that the replica tools might
> accidentally delete your whole file system (but only
> the files that you hadn't changed) if sources all of
> a sudden claims that the files are gone, like it did
> a few days ago.  It's trying to stay in sync with
> sources, after all.  This was a case that I would
> never have imagined, and it probably needs some
> sanity checking.

maybe i'm missing something.

all the files i checked were marked deleted in the log.
i think the problem was during database generation,
there was no connection to venti.  my patch should
address that.  i found it necessary when doing the
history transfer.  bad wireless connection.

maybe there's also a problem with applylog, but i didn't
find it.

- erik

Re: [9fans] Sources Gone?

[gas]

What if the replica/pull script defaulted to start with archiving to venti and print simple instruction how to restore in case of epic failure?

On the other hand, the -n flag works for me.

/G.


      __________________________________________________________
Går det långsamt? Skaffa dig en snabbare bredbandsuppkoppling. 
Sök och jämför priser hos Kelkoo.
http://www.kelkoo.se/c-100015813-bredband.html?partnerId=96914325

Re: [9fans] Sources Gone?

[Russ Cox]

On Thu, Jan 29, 2009 at 9:30 AM, erik quanstrom <quanstro@quanstro.net> wrote:
>> I can easily believe that the replica tools might
>> accidentally delete your whole file system (but only
>> the files that you hadn't changed) if sources all of
>> a sudden claims that the files are gone, like it did
>> a few days ago.  It's trying to stay in sync with
>> sources, after all.  This was a case that I would
>> never have imagined, and it probably needs some
>> sanity checking.
>
> maybe i'm missing something.
>
> all the files i checked were marked deleted in the log.
> i think the problem was during database generation,
> there was no connection to venti.  my patch should
> address that.  i found it necessary when doing the
> history transfer.  bad wireless connection.
>
> maybe there's also a problem with applylog, but i didn't
> find it.

it's very likely you're right.
i was just speculating based on knowledge
of the code.  i didn't look into the recent
failure at all.

russ

Re: [9fans] Sources Gone?

[Roman V. Shaposhnik]

On Thu, 2009-01-29 at 09:18 -0800, Russ Cox wrote:
> Onr fundamental difference is that the latter set is
> intended to keep trees exactly in sync,

"trees" tend to be highly overloaded, but if you refer
to the filesystem hierarchy as seem by open, then the
above statement, if applied to Git, is misleading.

Git specifically opted out for some additional complexity
in favor of treating the non-mutable hash addressed history
and write buffer differently.

Not realizing that git makes this distinction, makes people
surprised when they try to push their own history into
the repositories of others only to find out that the
tree they see locally doesn't match the tree that is on
the other end.

This rigid distinction was one of the factors that swayed
us in favor of Git when we did internal evaluation of
DSCMs about a yer ago. It really makes things much more
manageable.

> The replica tools are not SHA1-based because they
> cannot depend on the user having a venti to manage
> the blocks, and managing a separate copy of the data
> (like the dvcs's do) seemed out of character with
> fitting well into the surrounding system.

That is well understood and replica definitely has its
place in Plan9.

That said, what would be your thoughts on developing the
tools (and interfaces perhaps) for fetching up venti
blocks between two systems in a secure and manageable way.

It seems that as a *complimentary* solution this would
achieve quite a few benefits of Git/etc.

Thanks,
Roman.

Re: [9fans] Sources Gone?

[erik quanstrom]

> That said, what would be your thoughts on developing the
> tools (and interfaces perhaps) for fetching up venti
> blocks between two systems in a secure and manageable way.

i think this harks back to ye olde dump.  the main difference
is that "inode number" has now become a "score" and gotten
much fatter.

the problem at this level is it's hard to maintain any differences.
(you can't skip one of a sequence of 6 incremental dumps.)
or apply changes from one venti to another.  i don't see how
to make sense of venti-level syncing unless it is so basic that
you dump all the blocks from one venti into another.  in which
case, no special tools are needed.

- erik

Re: [9fans] Sources Gone?

[Russ Cox]

On Thu, Jan 29, 2009 at 1:09 PM, Roman V. Shaposhnik <rvs@sun.com> wrote:
> "trees" tend to be highly overloaded, but if you refer
> to the filesystem hierarchy as seem by open, then the
> above statement, if applied to Git, is misleading.

What I mean is that if there is some file in the "repository"
and I have a long-standing change to it (perhaps I have
edited a template config file), the revision control systems
tend to get annoyed that you're not checking that change back in.
Mercurial, for instance, requires you to say -f on hg merge
if you have pending changes.  And I think that might even
have disappeared in a recent version: you just can't merge
anymore in an "unclean" repository.

I am not talking about trees as trees of revision history;
I just meant the local file system.

I don't know how well Git handles this; I apologize for that.

I think using venti as a backend for Git would buy you
very little.  Git already does a good job of managing its
blocks.

Russ

Re: [9fans] Sources Gone?

[Roman V. Shaposhnik]

On Thu, 2009-01-29 at 14:33 -0800, Russ Cox wrote:
> On Thu, Jan 29, 2009 at 1:09 PM, Roman V. Shaposhnik <rvs@sun.com> wrote:
> I don't know how well Git handles this; I apologize for that.

Git doesn't get annoyed. In fact, with things like git stash you can
even test incremental changes to the merge without loosing the state
of your actual filesystem tree.

> I think using venti as a backend for Git would buy you
> very little.  Git already does a good job of managing its
> blocks.

Perhaps, I haven't made myself clear. What I asked you about
had nothing to do with Git. I was suggesting that letting two
venti systems efficiently trade blocks would be a desirable
addition to the system.

There was a question on this list not long time ago whether
getting access to venti blocks of the sources would be possible.
The answer at the time was "no". This is understandable since
the stock venti doesn't really offer any kind of security
mechanism for doing that.

However, the very fact that somebody else asked that question suggests
that the feature is not useless one.

Thanks,
Roman.

Re: [9fans] Sources Gone?

[Roman V. Shaposhnik]

On Thu, 2009-01-29 at 16:42 -0500, erik quanstrom wrote:
> > That said, what would be your thoughts on developing the
> > tools (and interfaces perhaps) for fetching up venti
> > blocks between two systems in a secure and manageable way.
>
> i think this harks back to ye olde dump.  the main difference
> is that "inode number" has now become a "score" and gotten
> much fatter.
>
> the problem at this level is it's hard to maintain any differences.
> (you can't skip one of a sequence of 6 incremental dumps.)

To use git as an example here, it has a concept of a shallow clone,
where it lets you skip the history older than a particular date.

> or apply changes from one venti to another.

What do you mean by "apply changes"? Each venti has a set of
blocks corresponding to vac hierarchies. Each vac hierarchy
has a single parent. Retrieving the missing blocks reachable
from a hierarchy of interest to you is all that is needed.

> i don't see how to make sense of venti-level syncing unless it is so
> basic that you dump all the blocks from one venti into another.
> in which case, no special tools are needed.

Some level of smartness in how block traversal is made needs to
be there. I would also like to have an agent close to the actual
server hosting venti archive being able to do things on my behalf
and only sending the results back (e.g. traversing the hierarchies
only to determine that certain things are not needed is painful
over the network). Plus there has to be a security mechanism that
would not allow me to request random blocks, unless I'm authorized.

Thanks,
Roman.

Re: [9fans] Sources Gone?

[Russ Cox]

> There was a question on this list not long time ago whether
> getting access to venti blocks of the sources would be possible.
> The answer at the time was "no". This is understandable since
> the stock venti doesn't really offer any kind of security
> mechanism for doing that.
>
> However, the very fact that somebody else asked that question suggests
> that the feature is not useless one.

It's certainly not useless.  Lots of interesting systems,
Git included, have been built on such arrangements.
It usually makes more sense in those systems to isolate
the blocks in question into their own storage place,
exactly so that you can avoid those security issues.

The problem with sources venti is that the venti server
that handles sources handles a collection of servers,
and there is no easy way in the venti model to only
allow access to the blocks used within the public
sources filesystem.

Russ

Re: [9fans] Sources Gone?

[erik quanstrom]

> What do you mean by "apply changes"? Each venti has a set of
> blocks corresponding to vac hierarchies. Each vac hierarchy
> has a single parent. Retrieving the missing blocks reachable
> from a hierarchy of interest to you is all that is needed.

you can put anything in venti.  they don't need to be vac
heirarchies.

> Some level of smartness in how block traversal is made needs to
> be there. I would also like to have an agent close to the actual
> server hosting venti archive being able to do things on my behalf
> and only sending the results back (e.g. traversing the hierarchies
> only to determine that certain things are not needed is painful
> over the network). Plus there has to be a security mechanism that
> would not allow me to request random blocks, unless I'm authorized.

after adding the needed features, what would the advantage
be over replica?

- erik

Re: [9fans] Sources Gone?

[Russ Cox]

> after adding the needed features, what would the advantage
> be over replica?

i'm fairly certain this discussion has diverged off
of replica and is now about just whether you could
build an interesting dvcs with venti as a low-level
building block, and if so, how.

russ

Re: [9fans] Sources Gone?

[lucio]

> i don't see how
> to make sense of venti-level syncing unless it is so basic that
> you dump all the blocks from one venti into another.  in which
> case, no special tools are needed.

Granted, but then security becomes an issue.  Perhaps not regarding
replica, but in the universal context.

++L

[9fans] Venti and version control (Was: Sources Gone?)

[lucio]

> whether you could
> build an interesting dvcs with venti as a low-level
> building block, and if so, how.

Well, the first suggestion may be how to partition venti, or is that a
silly starting point?  Security definitely defeats, or at least
dilutes some of venti's strengths.

++L

Re: [9fans] Sources Gone?

[lucio]

> Some level of smartness in how block traversal is made needs to
> be there.

That involves partitioning, which defeats the fundamental mechanics of
venti.  It then becomes preferable to run distinct venti services,
which is the only way in which different backing stores can be used at
this stage.  And as far as I can establish there is no way of
specifying distinct venti archives to a single fossil server, so now
you also need distinct fossils.  Feasible, of course, but a bit _too_
distributed to be practical.

++L

Re: [9fans] Sources Gone?

[Bruce Ellis]

sauces ain't gone anywhere. my kitchen is still full of them. fluffy
just had soy and honey on his pasta and pate. and i can 9fs sources at
the same time.

just my 3 euros worth.

brucee

On Fri, Jan 30, 2009 at 4:18 PM,  <lucio@proxima.alt.za> wrote:
>> Some level of smartness in how block traversal is made needs to
>> be there.
>
> That involves partitioning, which defeats the fundamental mechanics of
> venti.  It then becomes preferable to run distinct venti services,
> which is the only way in which different backing stores can be used at
> this stage.  And as far as I can establish there is no way of
> specifying distinct venti archives to a single fossil server, so now
> you also need distinct fossils.  Feasible, of course, but a bit _too_
> distributed to be practical.
>
> ++L
>
>
>

Re: [9fans] Sources Gone?

[Akshat Kumar]

Ground Control to Major Bruce:
I have the Acer Aspire One within reach.
Do you have any specific plans?


check ignitions,
ak

Re: [9fans] Sources Gone?

[Bruce Ellis]

Major Tom to Planet Earth.

I have no problems with tiny PCs. The eeePCs are still the best bet.
My Hellenic Lapdog is going to Brazil with me in a few hours.

brucee

On Sun, Feb 1, 2009 at 5:12 AM, Akshat Kumar
<akumar@mail.nanosouffle.net> wrote:
> Ground Control to Major Bruce:
> I have the Acer Aspire One within reach.
> Do you have any specific plans?
>
>
> check ignitions,
> ak
>
>

Re: [9fans] Sources Gone?

[Roman V. Shaposhnik]

On Fri, 2009-01-30 at 07:18 +0200, lucio@proxima.alt.za wrote:
> > Some level of smartness in how block traversal is made needs to
> > be there.
>
> That involves partitioning, which defeats the fundamental mechanics of
> venti.

I don't think it does. At least not in a way that is obvious to me.
The one and only fundamental limitation of the current interface
offered by venti is that I can give it a score to something that
doesn't belong to me and it gives me the information back. It is
the limitation of the API, not the way data is managed. IOW, if
a block that I genuinely own happens to also be referenced from
a hierarchy that I do NOT have access to -- its ok.

> It then becomes preferable to run distinct venti services,
> which is the only way in which different backing stores can be used at
> this stage.

Hm. I guess I need to understand what is the problem you seem to
be worried about.

Thanks,
Roman.

Re: [9fans] Sources Gone?

[erik quanstrom]

> I don't think it does. At least not in a way that is obvious to me.
> The one and only fundamental limitation of the current interface
> offered by venti is that I can give it a score to something that
> doesn't belong to me and it gives me the information back. It is
> the limitation of the API, not the way data is managed. IOW, if
> a block that I genuinely own happens to also be referenced from
> a hierarchy that I do NOT have access to -- its ok.

ownership doesn't mean anything at the venti level.  it really
is just a virtual disk drive with lba80 content addressing.
one doesn't "own" blocks on a regular disk drive, either.

suspending the preceeding logic for a bit, supposing that
you did "track ownership", then

each block could have any number of owners.  this would
mean that you couldn't store n copies of a block for
the cost of one block's storage.  you would need to
allocate some storage for each time the block is stored
to track ownership.

- erik

Re: [9fans] Sources Gone?

[Roman V. Shaposhnik]

On Mon, 2009-02-02 at 17:43 -0500, erik quanstrom wrote:
> > I don't think it does. At least not in a way that is obvious to me.
> > The one and only fundamental limitation of the current interface
> > offered by venti is that I can give it a score to something that
> > doesn't belong to me and it gives me the information back. It is
> > the limitation of the API, not the way data is managed. IOW, if
> > a block that I genuinely own happens to also be referenced from
> > a hierarchy that I do NOT have access to -- its ok.
>
> ownership doesn't mean anything at the venti level.  it really
> is just a virtual disk drive with lba80 content addressing.
> one doesn't "own" blocks on a regular disk drive, either.

Depends on how you look at it. From the drive's perspective -- you're
right. Nobody owns blocks. However, if a certain block happens
to be part of a filesystems that uses this particular drive then
the ownership can and will be tracked.

> suspending the preceeding logic for a bit, supposing that
> you did "track ownership", then

No need to suspend the logic and run this thought experiment. I have
no interest in assigning ACL to blocks.

That's why I said that the API that venti currently has is ill-suited
for the kind of public usage I have in mind. That doesn't mean that it
should be replaced or mutilated. It simply means firewalling in a spirit
of venti/ro

The proxy API will have to track the ownership. And it is very likely to
be more hierarchy-oriented, than stand-alone blocks-oriented.

Thanks,
Roman.

Re: [9fans] Sources Gone?

[erik quanstrom]

> Depends on how you look at it. From the drive's perspective -- you're
> right. Nobody owns blocks. However, if a certain block happens
> to be part of a filesystems that uses this particular drive then
> the ownership can and will be tracked.

the problem comes in the fact that as far as venti is concerned, there
is no filesystem.  the heirarchy bits are done in libventi, so

> The proxy API will have to track the ownership. And it is very likely to
> be more hierarchy-oriented, than stand-alone blocks-oriented.

protecting your root notes boils down to a matter of protecting
your root scores, since we assume that 2^80 is big enough to
deter guessing.  i don't see how a proxy will add security.
could you explain what attack vector you're worried about?

i would think that the least secure part of this is the unencrypted
transmission.

- erik

Re: [9fans] Sources Gone?

[lucio]

> The one and only fundamental limitation of the current interface
> offered by venti is that I can give it a score to something that
> doesn't belong to me and it gives me the information back. It is
> the limitation of the API, not the way data is managed.

I'm not sure how you'd fix this.  What if only a portion of the block
belongs to me and the other happens to be the password file?  You need
to draw boundaries at different security entities and that makes
compression and especially duplicate suppression much less effective.

You could argue that the security information provides its own
boundaries, but making venti aware of them does add a lot of
complications that in my opinion should be (and already are) solved
elsewhere.  I did propose that compression and encryption ought to be
concurrent in venti, but that solves only part of the problem and
raises new problems such as key management.

++L

PS: As for the problem we're trying to solve, I'm afraid I've long ago
lost track :-)

Re: [9fans] Sources Gone?

[erik quanstrom]

> > The one and only fundamental limitation of the current interface
> > offered by venti is that I can give it a score to something that
> > doesn't belong to me and it gives me the information back. It is
> > the limitation of the API, not the way data is managed.
>
> I'm not sure how you'd fix this.  What if only a portion of the block
> belongs to me and the other happens to be the password file?

venti just stores whole blocks.

if it did as you suggest, that would be a neat trick.
the arenas would reach a maximum size of 256 bytes.
(assuming you stored unique bytes).  everything else
would be index.  but, oh, what an index that'd be.

> You need to draw boundaries at different security entities and that makes
> compression and especially duplicate suppression much less effective.

i'm repeating myself, but that's nothing new.
i don't think you do need to draw boundraries.

it's turtles all the way UP.  given an arbitrarly large
tree, a single change to any leaf will propagate
all the way up creating a new path to the root.
in this respect, libventi trees are very similar to
ken's fs.

the only assumption i'm making is that eactly this
tree has not been stored before.  so really, if you
don't give out the root score, venti won't give you
up.

- erik

Re: [9fans] Sources Gone?

[lucio]

>> I'm not sure how you'd fix this.  What if only a portion of the block
>> belongs to me and the other happens to be the password file?
>
> venti just stores whole blocks.

Yes, but the content isn't guaranteed to be from a single user.  In
fact, venti has no clue.  Change that and it's not venti anymore.

++L

Re: [9fans] Sources Gone?

[Richard Miller]

> ownership doesn't mean anything at the venti level.  it really
> is just a virtual disk drive with lba80 content addressing.

I think you mean lba160.

Re: [9fans] Sources Gone?

[erik quanstrom]

> >> I'm not sure how you'd fix this.  What if only a portion of the block
> >> belongs to me and the other happens to be the password file?
> >
> > venti just stores whole blocks.
>
> Yes, but the content isn't guaranteed to be from a single user.  In
> fact, venti has no clue.  Change that and it's not venti anymore.

exactly.  but it's important to note that it's crypto hard to guess
somebody else's block.  since blocks are addressed by content, you
can't share a block with someone else unless both of you stored
the same block.  now if you are worried about libventi blocks with
pointers to other blocks, the same logic applies.  venti really doesn't
care what you store.

- erik

Re: [9fans] Sources Gone?

[roger peppe]

in the past i've pondered, in my crypto-naive way, if it
might be possible to make venti (or at least vac) somewhat
more secure by applying some kind of crypto to the
data structures containing scores.

to my mind, the biggest security vulnerability in venti
is the ability to unconditionally enumerate an entire file tree given
its root score. if the VtPointer data structures, or the
scores within them, were encrypted somehow, maybe
that vulnerability could be mitigated. scores would still
be useful, but only in conjunction with a (salted) key.

of course, this would mean that pointer blocks would no longer
be shared between file trees, but it's my suspicion that
they don't use a significant percentage of overall storage.

this wouldn't require a change to venti itself.

but as i said, i'm naive when it comes to crypto; maybe
there's no way of doing this with any decent degree
of security or usefulness.


2009/2/3 erik quanstrom <quanstro@quanstro.net>:
>> >> I'm not sure how you'd fix this.  What if only a portion of the block
>> >> belongs to me and the other happens to be the password file?
>> >
>> > venti just stores whole blocks.
>>
>> Yes, but the content isn't guaranteed to be from a single user.  In
>> fact, venti has no clue.  Change that and it's not venti anymore.
>
> exactly.  but it's important to note that it's crypto hard to guess
> somebody else's block.  since blocks are addressed by content, you
> can't share a block with someone else unless both of you stored
> the same block.  now if you are worried about libventi blocks with
> pointers to other blocks, the same logic applies.  venti really doesn't
> care what you store.
>
> - erik
>
>

Re: [9fans] Sources Gone?

[erik quanstrom]

> to my mind, the biggest security vulnerability in venti
> is the ability to unconditionally enumerate an entire file tree given
> its root score. if the VtPointer data structures, or the
> scores within them, were encrypted somehow, maybe
> that vulnerability could be mitigated. scores would still
> be useful, but only in conjunction with a (salted) key.

i'm not sure i understand.  either you have the key (score)
and you can decrypt the whole cyphertext (read the file tree
below), or you don't.  assuming of course that scores are too
hard to guess.  so the solution is: don't give out the root score.

(ot: you could think of a venti tree as a keyring, but that's
just nutty.)

> of course, this would mean that pointer blocks would no longer
> be shared between file trees, but it's my suspicion that
> they don't use a significant percentage of overall storage.

is there any other way to end up with the same pointer block
than starting with the same data?  conversely if either
data anywhere "below" (forgive the imprecision), pointer
blocks will change all the way up to the root and the root
will not be shared.  i don't see how information could leak,
either.

am i missing something?

- erik

Re: [9fans] Sources Gone?

[Anthony Sorace]

erik wrote:
> i'm not sure i understand.  either you have the key (score)
> and you can decrypt the whole cyphertext (read the file tree
> below), or you don't.  assuming of course that scores are too
> hard to guess.  so the solution is: don't give out the root score.

my read on the utility of rog's proposal is that you could then
pre-exchange the crypto key via secure channel (real live handoff or
whatnot) and then send root scores around freely over things like
email. unauthorized parties reading your email then don't get your
venti data.

the scheme has the advantage of being minimally intrusive, but it does
seem to be like putting the fix in the wrong place. i'd rather see an
authenticated connection mechanism, which would likely require more
changes (how do you store accounts and credentials? how do you feed
them to things like a fossil at boot?), but would have the same
benefits and more (i'd like to provide some clients read-only access,
for example).

Re: [9fans] Sources Gone?

[erik quanstrom]

> my read on the utility of rog's proposal is that you could then
> pre-exchange the crypto key via secure channel (real live handoff or
> whatnot) and then send root scores around freely over things like
> email. unauthorized parties reading your email then don't get your
> venti data.

if you want users, groups and access control, isn't the fs the
place to go?  i'm trying to see how doing fsey things at the
venti level would be useful, but i don't see it yet.

> the scheme has the advantage of being minimally intrusive, but it does
> seem to be like putting the fix in the wrong place. i'd rather see an
> authenticated connection mechanism, which would likely require more
> changes (how do you store accounts and credentials? how do you feed
> them to things like a fossil at boot?), but would have the same
> benefits and more (i'd like to provide some clients read-only access,
> for example).

i don't see the need for accounts or credentials (again, i think
the fs is there to provide those things).  but tls would be a good
idea if you plan on venti access across any potentially hostile
network.  you can steal scores off the wire.  i also don't see why
it wouldn't be trivial to add a venti-tls port and have venti just
push tls.

am i still missing something?

- erik

Re: [9fans] Sources Gone?

[roger peppe]

2009/2/3 erik quanstrom <quanstro@quanstro.net>:
>> to my mind, the biggest security vulnerability in venti
>> is the ability to unconditionally enumerate an entire file tree given
>> its root score. if the VtPointer data structures, or the
>> scores within them, were encrypted somehow, maybe
>> that vulnerability could be mitigated. scores would still
>> be useful, but only in conjunction with a (salted) key.
>
> i'm not sure i understand.  either you have the key (score)
> and you can decrypt the whole cyphertext (read the file tree
> below), or you don't.  assuming of course that scores are too
> hard to guess.  so the solution is: don't give out the root score.

i'm suggesting that the venti blocks containing pointers
(which are well separated, by design) could be stored encrypted.
so given a score, you can get a block out of venti
(as now), but you need the secret key in order to be able
to decrypt the scores contained within it. thus knowing a root score
without knowing the secret key does not enable browsing the entire tree.

obviously, you could do this for data blocks too, but then you
don't get any sharing at all.

> is there any other way to end up with the same pointer block
> than starting with the same data?

of course - two identical subtrees share the same pointer blocks,
but don't necessarily share the same root.

> i don't see how information could leak,

information can't leak in principle, but root scores are dangerous, which
is why open-access venti servers are problematic - if such a score
*does* happen to leak, then unconditional access to all your data has
also leaked.

i guess my proposal really just boils down to a way to be able
to write down scores in a not-entirely-secret place without
compromising everything.

> if you want users, groups and access control, isn't the fs the
> place to go?  i'm trying to see how doing fsey things at the
> venti level would be useful, but i don't see it yet.

the attraction, for me at any rate, is that certain operations are
really cheap and easy in venti, but expensive in the fs.
cloning/copying a multi-gigabyte tree being the canonical example.

Re: [9fans] Sources Gone?

[erik quanstrom]

> information can't leak in principle, but root scores are dangerous, which
> is why open-access venti servers are problematic - if such a score
> *does* happen to leak, then unconditional access to all your data has
> also leaked.

what prevents using a non-root score in a similar fashion?

- erik

Re: [9fans] Sources Gone?

[Brian L. Stuart]

> information can't leak in principle, but root scores are dangerous, which
> is why open-access venti servers are problematic - if such a score
> *does* happen to leak, then unconditional access to all your data has
> also leaked.

If I understand correctly, this line of discussion
is primarily motivated by the idea of an open-access
venti server.  And it looks to me like we're basically
getting to the point where we're recognizing that
to make that happen would require some very deep
changes to venti and it's underlying concepts.  It
sounds like a perfect place for an intermediary
server.  The venti itself doesn't need to be open-
access if there's a proxy server that is.  The proxy
can communicate with the clients using any unique
identifying key.  It doesn't have to be the same as
the score the back-end venti uses.  And the proxy
can do any kind of authentication you want it to.

Maybe I'm misunderstanding the problem we're trying
to solve, but if the objective is to provide open
venti access but the necessary protection mechanisms
really belong elsewhere, it seems reasonable to
create the elsewhere and not incorporate them into
venti.

Just a free observation (and worth every penny).

BLS

Re: [9fans] Sources Gone?

[lucio]

> but as i said, i'm naive when it comes to crypto; maybe
> there's no way of doing this with any decent degree
> of security or usefulness.

Encryption is always a two-way path: for every bit of piece of mind
encryption provides, there is a corresponding fear of losing access.
Also, encryption tends to slow things down and considering that I'm
now getting rather jaded about the fact that my fresh fossil/venti
server has taken five days, twice, to dump -a little over 1GB of data
on two separate occasions, I'm not sure encryption is affordable.

I think that venti should be protected, the recommended approach is to
place the fossil server and its venti archive on a separate physical
network (127.1 isn't too bad, but two hosts might be more convenient)
I'm not sure if one can do better than that.  It is unfortunate that
fossil and CPU services are very likely to occur on the same host.  In
that respect KenFS is a much nicer administrative entity.

++L

Re: [9fans] Sources Gone?

[lucio]

> i'm not sure i understand.  either you have the key (score)
> and you can decrypt the whole cyphertext (read the file tree
> below), or you don't.  assuming of course that scores are too
> hard to guess.  so the solution is: don't give out the root score.

fossil/last will find the most recent root score.  Has somebody
already mentioned that?

++L

Re: [9fans] Sources Gone?

[erik quanstrom]

> now getting rather jaded about the fact that my fresh fossil/venti
> server has taken five days, twice, to dump -a little over 1GB of data
> on two separate occasions, I'm not sure encryption is affordable.

i would imagine that cpu has nothing to do with it and encryption
would add no overhead at all.  i would image that seeks dominate
your performance numbers.

i don't think coraid could manage with such performance.  this
morning's dump is not outlandish but it is a bit bigger than
normal — 1.65gb in 8k blocks.  note that this isn't a super
i/o rate because kenfs doesn't go full-tilt boogy on dumps
and because the backup worm is across town.  but still,
it does manage to complete in under 12 minutes.

Tue Feb  3 04:01:43: automatic dump Tue Feb  3 05:00:06 EDT 2009
Tue Feb  3 04:01:43: sync: before dump
Tue Feb  3 04:01:43: cwroot 48903076->49095891
Tue Feb  3 04:01:43: sync: after cw
Tue Feb  3 04:01:43: roroot 48903079
Tue Feb  3 04:01:43: ->49095894 /2009/0203
Tue Feb  3 04:01:43: sync: after ro
Tue Feb  3 04:01:43: sblock 48773434->48903080 (->49095895)
Tue Feb  3 04:01:43: sync: all done
Tue Feb  3 04:01:43: 217056 blocks queued for worm
Tue Feb  3 04:01:43: 0 falsehits
Tue Feb  3 04:01:43: next dump at Wed Feb  4 05:00:00 EDT 2009
Tue Feb  3 04:13:35: 217056 blocks copied to worm

- erik

Re: [9fans] Sources Gone?

[sqweek]

On Tue, Feb 3, 2009 at 9:54 PM, erik quanstrom <quanstro@quanstro.net> wrote:
>> Yes, but the content isn't guaranteed to be from a single user.  In
>> fact, venti has no clue.  Change that and it's not venti anymore.
>
> exactly.  but it's important to note that it's crypto hard to guess
> somebody else's block.

 Is it? Well, to guess a specific block, obviously.
 I'm pretty ignorant about the structures used to store trees in venti
- would it be possible to reconstruct the block containing the root of
a particular tree given say, /n/dump?
 Presumably something along the lines of "vac /n/dump/2009/0204" would
suffice, but failing that you still don't need to guess exactly the
block you are looking for... How long would it take to brute force a
block of a tree (giving you references to lots of other blocks) from
venti?
-sqweek

[9fans] Some arithmetic [was: Re: Sources Gone?]

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 3489 bytes --]

On Wed, Feb 04, 2009 at 05:40:01PM +0900, sqweek wrote:
> On Tue, Feb 3, 2009 at 9:54 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> >> Yes, but the content isn't guaranteed to be from a single user.  In
> >> fact, venti has no clue.  Change that and it's not venti anymore.
> >
> > exactly.  but it's important to note that it's crypto hard to guess
> > somebody else's block.
> 
>  Is it? Well, to guess a specific block, obviously.
>  I'm pretty ignorant about the structures used to store trees in venti
> - would it be possible to reconstruct the block containing the root of
> a particular tree given say, /n/dump?

Presumably only if you could read all the data under /n/dump, in which case
there isn't a security risk.

>  Presumably something along the lines of "vac /n/dump/2009/0204" would
> suffice, but failing that you still don't need to guess exactly the
> block you are looking for... How long would it take to brute force a
> block of a tree (giving you references to lots of other blocks) from
> venti?

Assuming SHA-1 is indeed cryptographically secure (which is the assumption
made by the venti paper), you know only the type of the target block and no
bits of its score regardless of any partial information you know about the
block (total information obviously gives you the score).  Assuming you don't
care which block you read from the venti, and that the venti is storing K
blocks of the requisite type, the odds of you guessing the score of any
block stored is K/2^160.

If you're after data blocks and the venti is storing an exbibyte (2^60 bytes
== 2^47 8Ki blocks), I expect you'd have to take 2^113 queries to find your
first data block.

Assuming the venti is backing a fossil and has been running for 2^13 days
(roughly 22 years), there are 3*2^13 "root-like" scores stored (AFAIK: one
root for today's dump, one root of all past dumps, and one block that stores
both of these scores), so I expect you'd take 2^(147)/3 queries to find one.
Obviously some of these are more powerful than others, in terms of exposure,
so you might be relatively lucker or unluckier if you found a root block, in
which case you probably want to go buy as many lottery tickets as you can.

Given those odds, if somebody wants my vac scores, they'll break into my
office and steal the venti, or employ rubber hose cryptography.  Or maybe
SHA-1's really, really broken and has a much smaller output domain than
2^160...  in which case, somebody should write a version of venti that uses
one of the SHA2 variants or another hash.

If you need additional assurances, bear in mind that somewhere around 2^192
addition operations requires 32 years with a perfect Dyson sphere around the
sun and a thermodynamically perfect computer at 3.2K. Harnessing a typical
supernova gives 2^219 addition operations (Schneier, Applied Cryptography,
pp 158).  Assuming those figures are right, and that we lack a Dyson sphere
and there are no conveniently nearby supernovae, but that we can turn the
entire sun-facing solid angle of the earth into a similarly perfect
computer, we get 2^192/2^32*(4.5 x 10^(-10)) ~~ 2^129 addition operations in
a year (that magic number is the area of a circle with radius matching that
of the earth to the entire surface area of a sphere with radius one
astronomical unit).  That might be enough to find a data block with high
odds but not a root block under the above assumptions. :)

--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] Some arithmetic [was: Re: Sources Gone?]

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 480 bytes --]

On Wed, Feb 04, 2009 at 11:40:51AM -0500, Nathaniel W Filardo wrote:
> entire sun-facing solid angle of the earth into a similarly perfect
> computer, we get 2^192/2^32*(4.5 x 10^(-10)) ~~ 2^129 addition operations in

Rats, I got overly happy with exponentiation (should be 2^5, not 2^32).
Correcting the error gives 2^156 operations in a year, which is more than
sufficient to expect to find a root block.

Management would like to apologize for the oversight.
--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] Some arithmetic [was: Re: Sources Gone?]

[hiro]

> Assuming SHA-1 is indeed cryptographically secure (which is the assumption
> made by the venti paper)

Well, I read it like it was just sufficiently secure against
unintended collisions.
It's not intended to encrypt, but to efficiently store data.

Re: [9fans] Sources Gone?

[Roman V. Shaposhnik]

On Tue, 2009-02-03 at 17:30 +0000, Brian L. Stuart wrote:
> > information can't leak in principle, but root scores are dangerous, which
> > is why open-access venti servers are problematic - if such a score
> > *does* happen to leak, then unconditional access to all your data has
> > also leaked.
>
> If I understand correctly, this line of discussion
> is primarily motivated by the idea of an open-access
> venti server.

Correct. But with this caveat: I only care about
the blocks that are part of vac structures. Erik
keeps reminding us that venti doesn't care about
what's in the blocks. True. But now, I've drawn
a line. There's only one type of blocks that
I'm interested in -- blocks which are part of
vac structures.

> The venti itself doesn't need to be open-
> access if there's a proxy server that is.

Absolutely!

> Maybe I'm misunderstanding the problem we're trying
> to solve, but if the objective is to provide open
> venti access but the necessary protection mechanisms
> really belong elsewhere, it seems reasonable to
> create the elsewhere and not incorporate them into
> venti.

Looks like we're in a complete agreement. And thanks
for the summary!

Thanks,
Roman.

Re: [9fans] Some arithmetic [was: Re: Sources Gone?]

[Dave Eckhardt]

>> Assuming SHA-1 is indeed cryptographically secure (which is the
>> assumption made by the venti paper)
>
> Well, I read it like it was just sufficiently secure against
> unintended collisions.
>
> It's not intended to encrypt, but to efficiently store data.

While SHA-1 is indeed not intended to encrypt, it *is* intended
to be a secure hash (hence the name).  In order for it to do that
job, it must be computationally difficult for somebody to find
colliding material.  If it's "easy" to guess venti scores for
file-system roots, that suggests that SHA-1 systematically
doesn't cover certain parts of the output space.  If that is true,
that would be a big help for people trying to find collisions
(and, hence, forge signatures).  It could be that way, but a lot
of people are still acting in ways which will be painful if it is.

Said another way:  SHA-1 is designed to be a different kind of
"checksum" than CRC-32.  CRC's are designed to defend against
accidental corruption, but SHA-1 really is designed to make
deliberate collisions hard.

Dave Eckhardt

Re: [9fans] Some arithmetic [was: Re: Sources Gone?]

[Russ Cox]

Even if venti scores are completely unguessable,
using them as an authentication mechanism
is a mistake, because you can't change them.
It would be like having a fixed, unchangeable password
assigned to your account: once the password leaked
out into the world, one way or another, you'd have
no way to stop anyone on the internet from masquerading
as you or telling the password to others.

http://www.google.com/search?q="09+f9"

Russ

Re: [9fans] Some arithmetic [was: Re: Sources Gone?]

[erik quanstrom]

> http://www.google.com/search?q="09+f9"

is that a legal url?

- erik

Re: [9fans] Some arithmetic [was: Re: Sources Gone?]

[Roman V. Shaposhnik]

On Thu, 2009-02-05 at 12:41 -0500, erik quanstrom wrote:
> > http://www.google.com/search?q="09+f9"
>
> is that a legal url?

I don't think it is a legal URL, but most browsers
will turn it into a legal one before issuing a
GET request.

Thanks,
Roman.

P.S. Or am I missing some kind of a joke here? ;-)

Re: [9fans] Some arithmetic [was: Re: Sources Gone?]

[Micah Stetson]

>> > http://www.google.com/search?q="09+f9"
>>
>> is that a legal url?
>
> P.S. Or am I missing some kind of a joke here? ;-)

Intentional or not, it's a very good joke.

Micah

Re: [9fans] Some arithmetic [was: Re: Sources Gone?]

[Roman V. Shaposhnik]

On Thu, 2009-02-05 at 10:22 -0800, Micah Stetson wrote:
> >> > http://www.google.com/search?q="09+f9"
> >>
> >> is that a legal url?
> >
> > P.S. Or am I missing some kind of a joke here? ;-)
>
> Intentional or not, it's a very good joke.

but...but...erik always adds that look-i-am-using-plan9-smiley
to all of his jokes. i'm so confused...

Thanks,
Roman.

Re: [9fans] Some arithmetic [was: Re: Sources Gone?]

[erik quanstrom]

> but...but...erik always adds that look-i-am-using-plan9-smiley
> to all of his jokes. i'm so confused...

i do?  i guess ya learn something every day.

- erik

Re: [9fans] Some arithmetic [was: Re: Sources Gone?]

[hiro]

>> http://www.google.com/search?q="09+f9"
>
> is that a legal url?
>
> - erik

fortune worthy :D