[9fans] venti ports and productization

[9fans] venti ports and productization

[a]

a few times someone (ehg?) has talked about work
being done to get venti working on other OSs, and
i at least got the impression that this was
being donwe with an eye towards productization. i,
for one, am very interested. i've recently found
myself back in the world of "operations". we're
in the (slow) process of moving the IT stuff over
to Mac OS X from various vintages of Win32. we
currently use some commercial app for backups on
the server, and it's pretty good. but i'm
reluctant to buy more than one copy. it's not
*that* good. but given the equivalent of /n/dump
on OS X, and some decent management tools (so i
can have non-plan9 types admin it), i'd gladly
shell out $100/mac, more for the servers. i'd
even buy a few copies of a Win32 port, since it
may be a while until we've nixed all that crap.

so... is work on that progressing? is it more
research or product? any ETA? any details of the
work being done (just raw venti, or some level
of FS integration)? thanks,
ア

Re: [9fans] venti ports and productization

[jmk]

I'm not aware of any efforts here to 'productise' or port Venti.

Re: [9fans] venti ports and productization

[bs]

I think ehg talked/mentioned about productizing
factotum for (l)unix and windows, and IIRC not venti.
> a few times someone (ehg?) has talked about work
> being done to get venti working on other OSs, and
> i at least got the impression that this was
> being donwe with an eye towards productization. i,
> for one, am very interested. i've recently found
> myself back in the world of "operations". we're
> in the (slow) process of moving the IT stuff over
> to Mac OS X from various vintages of Win32. we
> currently use some commercial app for backups on
> the server, and it's pretty good. but i'm
> reluctant to buy more than one copy. it's not
> *that* good. but given the equivalent of /n/dump
> on OS X, and some decent management tools (so i
> can have non-plan9 types admin it), i'd gladly
> shell out $100/mac, more for the servers. i'd
> even buy a few copies of a Win32 port, since it
> may be a while until we've nixed all that crap.
> 
> so... is work on that progressing? is it more
> research or product? any ETA? any details of the
> work being done (just raw venti, or some level
> of FS integration)? thanks,
> ア

Re: [9fans] venti ports and productization

[Russ Cox]

A while ago I ported vac (not vacfs!) to Windows.
Dave and Eric made the port work a bit better, but
my impression (from afar) is that there hasn't been
any further work, especially not toward commercialization.
[Jmk's recent mail supports this.]

I ported Venti the server to FreeBSD a year ago, though
the port is incomplete and unsatisfactory for various reasons.
I also wrote a program to store FFS file system images
as vac-style streams and another program to parse the backed-up
images and serve them over an NFS loopback server.
We back up just under 200GB of disk every night.
I'm very happy with it.

amsterdam=; hist ~/.profile
Sep  6 22:50:53 EDT 2003 /home/am3/rsc/.profile 900
Sep  6 22:50:53 EDT 2003 /dump/am3/2003/1010/rsc/.profile 900
Apr 10 18:30:26 EDT 2003 /dump/am3/2003/0906/rsc/.profile 851
Jan 27 18:22:32 EST 2003 /dump/am3/2003/0410/rsc/.profile 797
amsterdam=;

These programs use the same unsatisfactory port environment.
Over the next few months I plan to clean them up and make them build
using the new port environment that I used for samterm (which
I'd be packaging right now if I weren't writing this email ;-).

I'm also thinking about various ways to speed up the Venti
server (which is currently dog slow) but again that won't happen
for a few months -- I'm up to my eyeballs in other things.

Russ

Re: [9fans] venti ports and productization

[a]

thanks, russ, this is what i was remembering. i will now
return to patiently awaiting your results. :-) if you're
interested (at any point), i've got a handfull of OS X
boxes i could help test with... or if you'd like an
account to work with... of course, i'll be grateful for
whatever you produce. FreeBSD and OS X are close enough
that i can generally make things work\0 (unless it's in
the kernel). thanks again,
ア

Re: [9fans] venti ports and productization

[Martin C.Atkins]

On Fri, 10 Oct 2003 20:37:01 -0400 a@9srv.net wrote:
> a few times someone (ehg?) has talked about work
> being done to get venti working on other OSs, and
>...

Well, perhaps now is a good time to own up! (Sorry for the
delay in responding, but we had to write a web-page first!)

I have been working on a re-implementation of Venti - initially for
Linux, but it should be easy to port the core functionality to other
Unix-like platforms, and eventually to Windows (I suppose...).

This is not a port of Venti, but a rewrite from scratch - I have not
seen/got the Venti source, although I have read the paper, and the
man pages. I'm using the same basic ideas of Venti, but also making a
number of different technical tradeoffs. Some of these trade-offs
still need to be validated in practice (I'm working on it! :-).

Before some of you get too excited - it isn't protocol-compatible
with Venti (at least, I don't think it is! That would be
too much of a coincidence! (see above) :-)

Our goal is to release this as a shareware-style product, hopefully
this year, for a price that would be consistent with your
$100/machine.

I hadn't put a Mac OS X version high on the agenda - should I
reconsider? :-)

See
	http://www.parvat.com/products/eternal
for more information.

Martin

--
Martin C. Atkins				martin@parvat.com
Parvat Infotech (Private) Limited		http://www.parvat.com

Re: [9fans] venti ports and productization

[a]

the fact that it's not protocol-compatable does lower (but not
totally eliminate) its value to me. if your version doesn't run
on Plan 9 and host fossil, that does, as well. on the other hand,
even the ideas are worth a lot, and if you can put file systems
on top of it (like fossil does, rather than just using it as an
archival vault) that my unix systems can use, that's a big win.

those first two points make me skeptical, but i'll be checking it
out today. fix those two (so why didn't you use the venti code?)
and provide a few tools my VP of Marketing can use, and get an
OS X port, and i'll buy a few coppies.
ア

Re: [9fans] venti ports and productization

[jmk]

On Tue Oct 21 06:27:39 EDT 2003, a@9srv.net wrote:
> ...
> those first two points make me skeptical, but i'll be checking it
> out today. fix those two (so why didn't you use the venti code?)
> and provide a few tools my VP of Marketing can use, and get an
> OS X port, and i'll buy a few coppies.
> ア

The feeling around here is that the Plan 9 Venti code is not a good
base with which to start a port. There may be a re-implementation at
some point, and it will be interesting to compare the tradeoffs each
version makes.

Re: [9fans] venti ports and productization

[a]

that's very interesting. my comment about "why rewrite" wasn't
intended to necessarily imply it should've been done that way,
although i realize it probably came off that way. what about
the current implementation is at issue? and (the big question)
is it expected that the new stuff will be protocol compatable?
ア

Re: [9fans] venti ports and productization

[Russ Cox]

> that's very interesting. my comment about "why rewrite" wasn't
> intended to necessarily imply it should've been done that way,
> although i realize it probably came off that way. what about
> the current implementation is at issue? and (the big question)
> is it expected that the new stuff will be protocol compatable?

The protocol is almost completely trivial.  There are only two
interesting messages -- read and write.  So ultimately it really
doesn't matter.

We have had problems with Venti since the Seans left,
but I am not convinced that we really need to start over
(perhaps I am the only one!).  We have had some problems
with the big Venti server, but my understanding is that
all of the problems appeared to be disk corruption rather
than software error.  I think the biggest problem is that
before the Seans left they didn't get the system scanning
itself for damage regularly.  Disks are going to fail and
we need to replace them, but we haven't been.  This does
contrast with the WORM, where the platters fail much less
frequently.

Then again, the amount of code involved is so small that
I do think that bulk alone doesn't justify sticking with
the current code if you want to start over.

If I started over, I suspect I would end up writing
substantially the same code, so I don't see the point.

Russ

Re: [9fans] venti ports and productization

[Rob Pike]

>
> Before some of you get too excited - it isn't protocol-compatible
> with Venti (at least, I don't think it is! That would be
> too much of a coincidence! (see above) :-)

you should make it protocol-compatible. can't be hard.
ask for the details.

> I hadn't put a Mac OS X version high on the agenda - should I
> reconsider? :-)
>
yes.

-rob

Re: [9fans] venti ports and productization

[rog]

[-- Attachment #1: Type: text/plain, Size: 205 bytes --]

> you should make it protocol-compatible. can't be hard.
> ask for the details.

i did most of a venti client-side module for inferno.

attached + documentation for anyone that might find it useful.

[-- Attachment #2: venti.tgz --]
[-- Type: application/x-tar, Size: 6246 bytes --]

Re: [9fans] venti ports and productization

[Scott Schwartz]

| the current implementation is at issue? and (the big question)
| is it expected that the new stuff will be protocol compatable?

That is the big question.  Independent but compatable implementations
is a sign of success.

Re: [9fans] venti ports and productization

[rog]

> Then again, the amount of code involved is so small that
> I do think that bulk alone doesn't justify sticking with
> the current code if you want to start over.

mind you, it looks like it could be somewhat smaller still.
there are various bits that look quite complex but in fact
are hardly used (e.g. /sys/src/libventi/packet.c)

Re: [9fans] venti ports and productization

[C H Forsyth]

[-- Attachment #1: Type: text/plain, Size: 53 bytes --]

that's obviously why microsoft is a complete failure!

[-- Attachment #2: Type: message/rfc822, Size: 2426 bytes --]

From: Scott Schwartz <schwartz@bio.cse.psu.edu>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] venti ports and productization
Date: Tue, 21 Oct 2003 13:20:50 -0400
Message-ID: <20031021172051.527.qmail@f.bio.cse.psu.edu>

| the current implementation is at issue? and (the big question)
| is it expected that the new stuff will be protocol compatable?

That is the big question.  Independent but compatable implementations
is a sign of success.

Re: [9fans] venti ports and productization

[Dan Cross]

> that's obviously why microsoft is a complete failure!

But what Scott mentioned is only *one* sign of success!

	- Dan C.

Re: [9fans] venti ports and productization

[okamoto]

Russ--

> I think the biggest problem is that
> before the Seans left they didn't get the system scanning
> itself for damage regularly.  Disks are going to fail and
> we need to replace them, but we haven't been.  This does
> contrast with the WORM, where the platters fail much less
> frequently.

Hmmm, somthing bad is happening in Bell-Labs...
Ok, from now, I'll read messages more carefully.
Don't forget, many of Plan 9 users are gratefull of your
efforts on Plan 9.   And also, I hope all the developping
team of Plan 9 re-united as before.  I understand many difficulties
around Lucent, which may indicate that it's not the time
to develope OS in a company alone anymore...  I'm not sure,
of course.

--Kenji

Re: [9fans] venti ports and productization

[okamoto]

Awawawa----!

Re: [9fans] venti ports and productization

[a]

// Awawawa----!

okay, my Japanese is a bit rusty (read: i only know about
half a dozen words) but i'm pretty sure that translates to
"Oh, so sorry. That was intended to go only to Russ, not
the entire list. I'm quite embarrassed."

marvelously efficient language.
ア

Re: [9fans] venti ports and productization

[Martin C.Atkins]

On Tue, 21 Oct 2003 09:05:31 -0700 Rob Pike <rob@mightycheese.com> wrote:
>
> you should make it protocol-compatible. can't be hard.
> ask for the details.

and On Tue, 21 Oct 2003 06:20:47 -0400 a@9srv.net wrote:
> the fact that it's not protocol-compatable does lower (but not
> totally eliminate) its value to me. if your version doesn't run
> on Plan 9 and host fossil, that does, as well. on the other hand,

I'll have a look at what it would take to make the protocol
compatible. Actually, one of the things I've chosen to do
differently makes the protocol incompatible (it changes the 'score',
in a way that I'd prefer not to discuss until I'm sure it is
desirable!) However, it might be possible to support the Venti
protocol simultaneously.

Rog's Inferno code looks like a good starting point for information
(thanks Rog!), although perhaps I could download the 'real thing'
now, without it influencing my ideas too much about how to approach
various design decisions. (I came across a 'Linux/BSD/etc CD' vendor
here in Bangalore last week, who listed Plan 9 as being available -
is there a Plan 9 UG in Bangalore?!)

> those first two points make me skeptical, but i'll be checking it
> out today. fix those two (so why didn't you use the venti code?)

I also wanted to be very certain that no Venti code found its
way into my implementation, lest I violate any copyrights.
Patents, are of course another issue. I have searched without finding
anything that seems relevant, but that is the problem with patents!

> even the ideas are worth a lot, and if you can put file systems
> on top of it (like fossil does, rather than just using it as an
> archival vault) that my unix systems can use, that's a big win.

I haven't any plans at the moment for a read-write filesystem, like
fossil, but I certainly aim to provide something like /backup.

> and provide a few tools my VP of Marketing can use, and get an
> OS X port, and i'll buy a few coppies.

We certainly intend that the system will be 'user-friendly', but in
the Unix sense, not the lots-of-icons Windows sense.

Martin

--
Martin C. Atkins				martin@parvat.com
Parvat Infotech (Private) Limited		http://www.parvat.com

Re: [9fans] venti ports and productization

[Richard Miller]

> We have had some problems
> with the big Venti server, but my understanding is that
> all of the problems appeared to be disk corruption rather
> than software error.

The software could do a bit better in recovering from hardware errors.
If a bad sector makes a venti block unreadable, rdarena and wrarena
could use the redundant information in the arena trailer to skip the
bad block and continue, but they just give up.  Worse, if checkarenas
(with -f) fails to read a block, instead of using the trailer information
to skip the block, it tries to correct the mismatch between block label
and trailer by copying garbage into the trailer entry from the buffer
it just failed to read into.  Oops.

-- Richard

Re: [9fans] venti ports and productization

[jmk]

On Wed Oct 22 10:58:46 EDT 2003, miller@hamnavoe.demon.co.uk wrote:
> > We have had some problems
> > with the big Venti server, but my understanding is that
> > all of the problems appeared to be disk corruption rather
> > than software error.
>
> The software could do a bit better in recovering from hardware errors.
> If a bad sector makes a venti block unreadable, rdarena and wrarena
> could use the redundant information in the arena trailer to skip the
> bad block and continue, but they just give up.  Worse, if checkarenas
> (with -f) fails to read a block, instead of using the trailer information
> to skip the block, it tries to correct the mismatch between block label
> and trailer by copying garbage into the trailer entry from the buffer
> it just failed to read into.  Oops.
>
> -- Richard

That's pretty much right, in my limited experience. When I started looking
at the errors we have I hacked up the arena read programme I have to do some
resynchronisation, it needs more work, though.
Looking at the errors we have, I don't think I'd ever want to write back
into the arena to try to fix it, that stuff with VtTypeCorrupt just seems
wrong (as does the definition of VtTypeCorrupt as VtMaxType).